Document (#34086)

Author

Bodoff, D.

Title

Test theory for evaluating reliability of IR test collections

Source

Information processing and management. 44(2008) no.3, S.1117-1145

Year

2008

Abstract

Classical test theory offers theoretically derived reliability measures such as Cronbach's alpha, which can be applied to measure the reliability of a set of Information Retrieval test results. The theory also supports item analysis, which identifies queries that are hampering the test's reliability, and which may be candidates for refinement or removal. A generalization of Classical Test Theory, called Generalizability Theory, provides an even richer set of tools. It allows us to estimate the reliability of a test as a function of the number of queries, assessors (relevance judges), and other aspects of the test's design. One novel aspect of Generalizability Theory is that it allows this estimation of reliability even before the test collection exists, based purely on the numbers of queries and assessors that it will contain. These calculations can help test designers in advance, by allowing them to compare the reliability of test designs with various numbers of queries and relevance assessors, and to spend their limited budgets on a design that maximizes reliability. Empirical analysis shows that in cases for which our data is representative, having more queries is more helpful for reliability than having more assessors. It also suggests that reliability may be improved with a per-document performance measure, as opposed to a document-set based performance measure, where appropriate. The theory also clarifies the implicit debate in IR literature regarding the nature of error in relevance judgments.

Similar documents (author)

1. Bodoff, D.: ¬A re-unification of two competing models for document retrieval (1999) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bodoff in 2951) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 2951, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=2951)
   

2. Bodoff, D.: Relevance for browsing, relevance for searching (2006) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bodoff in 4909) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 4909, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=4909)
   

3. Bodoff, D.: Emergence of terminological conventions as a searcher-indexer coordination game (2009) 5.66

   5.661144 = sum of:
     5.661144 = weight(author_txt:bodoff in 3299) [ClassicSimilarity], result of:
       5.661144 = fieldWeight in 3299, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.625 = fieldNorm(doc=3299)
   

4. Bodoff, D.; Kambil, A.: Partial coordination : I. The best of pre-coordination and post-coordination (1998) 4.53

   4.528915 = sum of:
     4.528915 = weight(author_txt:bodoff in 2322) [ClassicSimilarity], result of:
       4.528915 = fieldWeight in 2322, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.5 = fieldNorm(doc=2322)
   

5. Bodoff, D.; Kambil, A.: Partial coordination : II. A preliminary evaluation and failure analysis (1998) 4.53

   4.528915 = sum of:
     4.528915 = weight(author_txt:bodoff in 2323) [ClassicSimilarity], result of:
       4.528915 = fieldWeight in 2323, product of:
         1.0 = tf(freq=1.0), with freq of:
           1.0 = termFreq=1.0
         9.05783 = idf(docFreq=13, maxDocs=44218)
         0.5 = fieldNorm(doc=2323)
   

Similar documents (content)

1. Bodoff, D.; Wu, B.; Wong, K.Y.M.: Relevance data for language models using maximum likelihood (2003) 0.22

   0.22290234 = sum of:
     0.22290234 = product of:
       0.7960798 = sum of:
         0.029522913 = weight(abstract_txt:document in 1822) [ClassicSimilarity], result of:
           0.029522913 = score(doc=1822,freq=2.0), product of:
             0.05187423 = queryWeight, product of:
               1.058157 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.011420374 = queryNorm
             0.5691248 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.026202435 = weight(abstract_txt:performance in 1822) [ClassicSimilarity], result of:
           0.026202435 = score(doc=1822,freq=1.0), product of:
             0.060360104 = queryWeight, product of:
               1.1414299 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.011420374 = queryNorm
             0.43410188 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.015537282 = weight(abstract_txt:also in 1822) [ClassicSimilarity], result of:
           0.015537282 = score(doc=1822,freq=1.0), product of:
             0.048768118 = queryWeight, product of:
               1.2565734 = boost
               3.3983476 = idf(docFreq=4017, maxDocs=44218)
               0.011420374 = queryNorm
             0.31859508 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3983476 = idf(docFreq=4017, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.06716077 = weight(abstract_txt:relevance in 1822) [ClassicSimilarity], result of:
           0.06716077 = score(doc=1822,freq=2.0), product of:
             0.102711655 = queryWeight, product of:
               1.8236006 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011420374 = queryNorm
             0.65387684 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.08786428 = weight(abstract_txt:queries in 1822) [ClassicSimilarity], result of:
           0.08786428 = score(doc=1822,freq=1.0), product of:
             0.18353143 = queryWeight, product of:
               3.1470199 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.011420374 = queryNorm
             0.47874242 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.35391504 = weight(abstract_txt:assessors in 1822) [ClassicSimilarity], result of:
           0.35391504 = score(doc=1822,freq=1.0), product of:
             0.4313186 = queryWeight, product of:
               4.315075 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.011420374 = queryNorm
             0.820542 = fieldWeight in 1822, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
         0.21587712 = weight(abstract_txt:test in 1822) [ClassicSimilarity], result of:
           0.21587712 = score(doc=1822,freq=2.0), product of:
             0.3226419 = queryWeight, product of:
               5.598103 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.011420374 = queryNorm
             0.669092 = fieldWeight in 1822, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.09375 = fieldNorm(doc=1822)
       0.28 = coord(7/25)
   

2. Verberne, S.; Heijden, M. van der; Hinne, M.; Sappelli, M.; Koldijk, S.; Hoenkamp, E.; Kraaij, W.: Reliability and validity of query intent assessments (2013) 0.22

   0.21760777 = sum of:
     0.21760777 = product of:
       0.90669906 = sum of:
         0.012162629 = weight(abstract_txt:that in 1104) [ClassicSimilarity], result of:
           0.012162629 = score(doc=1104,freq=3.0), product of:
             0.047417063 = queryWeight, product of:
               1.7522748 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011420374 = queryNorm
             0.2565032 = fieldWeight in 1104, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.042425964 = weight(abstract_txt:measure in 1104) [ClassicSimilarity], result of:
           0.042425964 = score(doc=1104,freq=1.0), product of:
             0.124844074 = queryWeight, product of:
               2.010499 = boost
               5.437306 = idf(docFreq=522, maxDocs=44218)
               0.011420374 = queryNorm
             0.33983162 = fieldWeight in 1104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.437306 = idf(docFreq=522, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.05857619 = weight(abstract_txt:queries in 1104) [ClassicSimilarity], result of:
           0.05857619 = score(doc=1104,freq=1.0), product of:
             0.18353143 = queryWeight, product of:
               3.1470199 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.011420374 = queryNorm
             0.31916162 = fieldWeight in 1104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.40866587 = weight(abstract_txt:assessors in 1104) [ClassicSimilarity], result of:
           0.40866587 = score(doc=1104,freq=3.0), product of:
             0.4313186 = queryWeight, product of:
               4.315075 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.011420374 = queryNorm
             0.94748026 = fieldWeight in 1104, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.101765454 = weight(abstract_txt:test in 1104) [ClassicSimilarity], result of:
           0.101765454 = score(doc=1104,freq=1.0), product of:
             0.3226419 = queryWeight, product of:
               5.598103 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.011420374 = queryNorm
             0.315413 = fieldWeight in 1104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
         0.28310293 = weight(abstract_txt:reliability in 1104) [ClassicSimilarity], result of:
           0.28310293 = score(doc=1104,freq=1.0), product of:
             0.66100174 = queryWeight, product of:
               8.446184 = boost
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.011420374 = queryNorm
             0.42829376 = fieldWeight in 1104, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.0625 = fieldNorm(doc=1104)
       0.24 = coord(6/25)
   

3. Xu, Y.: Relevance judgment in epistemic and hedonic information searches (2007) 0.17

   0.17474848 = sum of:
     0.17474848 = product of:
       0.72811866 = sum of:
         0.017396543 = weight(abstract_txt:document in 64) [ClassicSimilarity], result of:
           0.017396543 = score(doc=64,freq=1.0), product of:
             0.05187423 = queryWeight, product of:
               1.058157 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.011420374 = queryNorm
             0.33536002 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.01091468 = weight(abstract_txt:which in 64) [ClassicSimilarity], result of:
           0.01091468 = score(doc=64,freq=1.0), product of:
             0.047899064 = queryWeight, product of:
               1.4379797 = boost
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.011420374 = queryNorm
             0.22786833 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.008777621 = weight(abstract_txt:that in 64) [ClassicSimilarity], result of:
           0.008777621 = score(doc=64,freq=1.0), product of:
             0.047417063 = queryWeight, product of:
               1.7522748 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011420374 = queryNorm
             0.18511525 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.1187246 = weight(abstract_txt:relevance in 64) [ClassicSimilarity], result of:
           0.1187246 = score(doc=64,freq=9.0), product of:
             0.102711655 = queryWeight, product of:
               1.8236006 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011420374 = queryNorm
             1.1559019 = fieldWeight in 64, product of:
               3.0 = tf(freq=9.0), with freq of:
                 9.0 = termFreq=9.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.07184527 = weight(abstract_txt:theory in 64) [ClassicSimilarity], result of:
           0.07184527 = score(doc=64,freq=1.0), product of:
             0.20273614 = queryWeight, product of:
               3.9135768 = boost
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.011420374 = queryNorm
             0.35437822 = fieldWeight in 64, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
         0.50045997 = weight(abstract_txt:reliability in 64) [ClassicSimilarity], result of:
           0.50045997 = score(doc=64,freq=2.0), product of:
             0.66100174 = queryWeight, product of:
               8.446184 = boost
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.011420374 = queryNorm
             0.75712353 = fieldWeight in 64, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.078125 = fieldNorm(doc=64)
       0.24 = coord(6/25)
   

4. Bodoff, D.; Wong, S.P.-S.: Documents and queries as random variables : history and implications (2006) 0.17

   0.16915251 = sum of:
     0.16915251 = product of:
       0.46986806 = sum of:
         0.017396543 = weight(abstract_txt:document in 193) [ClassicSimilarity], result of:
           0.017396543 = score(doc=193,freq=1.0), product of:
             0.05187423 = queryWeight, product of:
               1.058157 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.011420374 = queryNorm
             0.33536002 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.029170485 = weight(abstract_txt:even in 193) [ClassicSimilarity], result of:
           0.029170485 = score(doc=193,freq=1.0), product of:
             0.07321591 = queryWeight, product of:
               1.2571207 = boost
               5.0997415 = idf(docFreq=732, maxDocs=44218)
               0.011420374 = queryNorm
             0.3984173 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.0997415 = idf(docFreq=732, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.012990533 = weight(abstract_txt:more in 193) [ClassicSimilarity], result of:
           0.012990533 = score(doc=193,freq=1.0), product of:
             0.048875526 = queryWeight, product of:
               1.2579564 = boost
               3.402088 = idf(docFreq=4002, maxDocs=44218)
               0.011420374 = queryNorm
             0.2657881 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.402088 = idf(docFreq=4002, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.01091468 = weight(abstract_txt:which in 193) [ClassicSimilarity], result of:
           0.01091468 = score(doc=193,freq=1.0), product of:
             0.047899064 = queryWeight, product of:
               1.4379797 = boost
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.011420374 = queryNorm
             0.22786833 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.017555242 = weight(abstract_txt:that in 193) [ClassicSimilarity], result of:
           0.017555242 = score(doc=193,freq=4.0), product of:
             0.047417063 = queryWeight, product of:
               1.7522748 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011420374 = queryNorm
             0.3702305 = fieldWeight in 193, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.055967312 = weight(abstract_txt:relevance in 193) [ClassicSimilarity], result of:
           0.055967312 = score(doc=193,freq=2.0), product of:
             0.102711655 = queryWeight, product of:
               1.8236006 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011420374 = queryNorm
             0.5448974 = fieldWeight in 193, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.12682116 = weight(abstract_txt:queries in 193) [ClassicSimilarity], result of:
           0.12682116 = score(doc=193,freq=3.0), product of:
             0.18353143 = queryWeight, product of:
               3.1470199 = boost
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.011420374 = queryNorm
             0.6910051 = fieldWeight in 193, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.106586 = idf(docFreq=727, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.07184527 = weight(abstract_txt:theory in 193) [ClassicSimilarity], result of:
           0.07184527 = score(doc=193,freq=1.0), product of:
             0.20273614 = queryWeight, product of:
               3.9135768 = boost
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.011420374 = queryNorm
             0.35437822 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.5360413 = idf(docFreq=1287, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
         0.12720682 = weight(abstract_txt:test in 193) [ClassicSimilarity], result of:
           0.12720682 = score(doc=193,freq=1.0), product of:
             0.3226419 = queryWeight, product of:
               5.598103 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.011420374 = queryNorm
             0.39426625 = fieldWeight in 193, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.078125 = fieldNorm(doc=193)
       0.36 = coord(9/25)
   

5. Kutlu, M.; Elsayed, T.; Lease, M.: Intelligent topic selection for low-cost information retrieval evaluation : a new perspective on deep vs. shallow judging (2018) 0.16

   0.1647495 = sum of:
     0.1647495 = product of:
       0.5148422 = sum of:
         0.010437926 = weight(abstract_txt:document in 5092) [ClassicSimilarity], result of:
           0.010437926 = score(doc=5092,freq=1.0), product of:
             0.05187423 = queryWeight, product of:
               1.058157 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.011420374 = queryNorm
             0.20121601 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.007768641 = weight(abstract_txt:also in 5092) [ClassicSimilarity], result of:
           0.007768641 = score(doc=5092,freq=1.0), product of:
             0.048768118 = queryWeight, product of:
               1.2565734 = boost
               3.3983476 = idf(docFreq=4017, maxDocs=44218)
               0.011420374 = queryNorm
             0.15929754 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3983476 = idf(docFreq=4017, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.013500159 = weight(abstract_txt:more in 5092) [ClassicSimilarity], result of:
           0.013500159 = score(doc=5092,freq=3.0), product of:
             0.048875526 = queryWeight, product of:
               1.2579564 = boost
               3.402088 = idf(docFreq=4002, maxDocs=44218)
               0.011420374 = queryNorm
             0.2762151 = fieldWeight in 5092, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.402088 = idf(docFreq=4002, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.006548808 = weight(abstract_txt:which in 5092) [ClassicSimilarity], result of:
           0.006548808 = score(doc=5092,freq=1.0), product of:
             0.047899064 = queryWeight, product of:
               1.4379797 = boost
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.011420374 = queryNorm
             0.136721 = fieldWeight in 5092, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.9167147 = idf(docFreq=6503, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.010533145 = weight(abstract_txt:that in 5092) [ClassicSimilarity], result of:
           0.010533145 = score(doc=5092,freq=4.0), product of:
             0.047417063 = queryWeight, product of:
               1.7522748 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.011420374 = queryNorm
             0.22213829 = fieldWeight in 5092, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.033580385 = weight(abstract_txt:relevance in 5092) [ClassicSimilarity], result of:
           0.033580385 = score(doc=5092,freq=2.0), product of:
             0.102711655 = queryWeight, product of:
               1.8236006 = boost
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.011420374 = queryNorm
             0.32693842 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.931848 = idf(docFreq=866, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.13219719 = weight(abstract_txt:test in 5092) [ClassicSimilarity], result of:
           0.13219719 = score(doc=5092,freq=3.0), product of:
             0.3226419 = queryWeight, product of:
               5.598103 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.011420374 = queryNorm
             0.40973347 = fieldWeight in 5092, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
         0.30027598 = weight(abstract_txt:reliability in 5092) [ClassicSimilarity], result of:
           0.30027598 = score(doc=5092,freq=2.0), product of:
             0.66100174 = queryWeight, product of:
               8.446184 = boost
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.011420374 = queryNorm
             0.45427412 = fieldWeight in 5092, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.8527 = idf(docFreq=126, maxDocs=44218)
               0.046875 = fieldNorm(doc=5092)
       0.32 = coord(8/25)