Document (#39673)

Author

Kenter, T.
Balog, K.
Rijke, M. de

Title

Evaluating document filtering systems over time

Source

Information processing and management. 51(2015) no.6, S.791-808

Year

2015

Abstract

Document filtering is a popular task in information retrieval. A stream of documents arriving over time is filtered for documents relevant to a set of topics. The distinguishing feature of document filtering is the temporal aspect introduced by the stream of documents. Document filtering systems, up to now, have been evaluated in terms of traditional metrics like (micro- or macro-averaged) precision, recall, MAP, nDCG, F1 and utility. We argue that these metrics do not capture all relevant aspects of the systems being evaluated. In particular, they lack support for the temporal dimension of the task. We propose a time-sensitive way of measuring performance of document filtering systems over time by employing trend estimation. In short, the performance is calculated for batches, a trend line is fitted to the results, and the estimated performance of systems at the end of the evaluation period is used to compare systems. We detail the application of our proposed trend estimation framework and examine the assumptions that need to hold for valid significance testing. Additionally, we analyze the requirements a document filtering metric has to meet and show that traditional macro-averaged true-positive-based metrics, like precision, recall and utility fail to capture essential information when applied in a batch setting. In particular, false positives returned in a batch for topics that are absent from the ground truth in that batch go unnoticed. This is a serious flaw as over-generation of a system might be overlooked this way. We propose a new metric, aptness, that does capture false positives. We incorporate this metric in an overall score and show that this new score does meet all requirements. To demonstrate the results of our proposed evaluation methodology, we analyze the runs submitted to the two most recent editions of a document filtering evaluation campaign. We re-evaluate the runs submitted to the Cumulative Citation Recommendation task of the 2012 and 2013 editions of the TREC Knowledge Base Acceleration track, and show that important new insights emerge.

Content

Vgl.: doi:10.1016/j.ipm.2015.03.005.

Footnote

Beitrag in einem Themenschwerpunkt "Time and information retrieval"

Similar documents (author)

1. Balog, K.; Azzopardi, L.; Rijke, M. de: ¬A language modeling framework for expert finding (2009) 4.92

   4.921196 = sum of:
     4.921196 = sum of:
       1.9682486 = weight(author_txt:rijke in 2447) [ClassicSimilarity], result of:
         1.9682486 = score(doc=2447,freq=1.0), product of:
           0.606616 = queryWeight, product of:
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.07010985 = queryNorm
           3.2446368 = fieldWeight in 2447, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.375 = fieldNorm(doc=2447)
       2.9529476 = weight(author_txt:balog in 2447) [ClassicSimilarity], result of:
         2.9529476 = score(doc=2447,freq=1.0), product of:
           0.794995 = queryWeight, product of:
             1.1447885 = boost
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.07010985 = queryNorm
           3.7144227 = fieldWeight in 2447, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.375 = fieldNorm(doc=2447)
   

2. Hofmann, K.; Balog, K.; Bogers, T.; Rijke, M. de: Contextual factors for finding similar experts (2010) 4.10

   4.100997 = sum of:
     4.100997 = sum of:
       1.6402073 = weight(author_txt:rijke in 3456) [ClassicSimilarity], result of:
         1.6402073 = score(doc=3456,freq=1.0), product of:
           0.606616 = queryWeight, product of:
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.07010985 = queryNorm
           2.703864 = fieldWeight in 3456, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.3125 = fieldNorm(doc=3456)
       2.4607897 = weight(author_txt:balog in 3456) [ClassicSimilarity], result of:
         2.4607897 = score(doc=3456,freq=1.0), product of:
           0.794995 = queryWeight, product of:
             1.1447885 = boost
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.07010985 = queryNorm
           3.0953524 = fieldWeight in 3456, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.3125 = fieldNorm(doc=3456)
   

3. Berendsen, R.; Rijke, M. de; Balog, K.; Bogers, T.; Bosch, A. van den: On the assessment of expertise profiles (2013) 3.28

   3.2807975 = sum of:
     3.2807975 = sum of:
       1.3121657 = weight(author_txt:rijke in 1089) [ClassicSimilarity], result of:
         1.3121657 = score(doc=1089,freq=1.0), product of:
           0.606616 = queryWeight, product of:
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.07010985 = queryNorm
           2.1630912 = fieldWeight in 1089, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             8.652365 = idf(docFreq=20, maxDocs=44218)
             0.25 = fieldNorm(doc=1089)
       1.9686317 = weight(author_txt:balog in 1089) [ClassicSimilarity], result of:
         1.9686317 = score(doc=1089,freq=1.0), product of:
           0.794995 = queryWeight, product of:
             1.1447885 = boost
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.07010985 = queryNorm
           2.476282 = fieldWeight in 1089, product of:
             1.0 = tf(freq=1.0), with freq of:
               1.0 = termFreq=1.0
             9.905128 = idf(docFreq=5, maxDocs=44218)
             0.25 = fieldNorm(doc=1089)
   

4. De Rijke, M. -> Rijke, M. de: 1.39

   1.3917619 = sum of:
     1.3917619 = product of:
       2.7835238 = sum of:
         2.7835238 = weight(author_txt:rijke in 4116) [ClassicSimilarity], result of:
           2.7835238 = score(doc=4116,freq=2.0), product of:
             0.606616 = queryWeight, product of:
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.07010985 = queryNorm
             4.588609 = fieldWeight in 4116, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.375 = fieldNorm(doc=4116)
       0.5 = coord(1/2)
   

5. Li, X.; Rijke, M.de: Characterizing and predicting downloads in academic search (2019) 1.31

   1.3121657 = sum of:
     1.3121657 = product of:
       2.6243315 = sum of:
         2.6243315 = weight(author_txt:rijke in 5103) [ClassicSimilarity], result of:
           2.6243315 = score(doc=5103,freq=1.0), product of:
             0.606616 = queryWeight, product of:
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.07010985 = queryNorm
             4.3261824 = fieldWeight in 5103, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.652365 = idf(docFreq=20, maxDocs=44218)
               0.5 = fieldNorm(doc=5103)
       0.5 = coord(1/2)
   

Similar documents (content)

1. Sakai, T.: On the reliability of information retrieval metrics based on graded relevance (2007) 0.16

   0.15825212 = sum of:
     0.15825212 = product of:
       0.65938383 = sum of:
         0.08203149 = weight(abstract_txt:submitted in 910) [ClassicSimilarity], result of:
           0.08203149 = score(doc=910,freq=1.0), product of:
             0.14988182 = queryWeight, product of:
               1.0410807 = boost
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.020550529 = queryNorm
             0.5473078 = fieldWeight in 910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0055394 = idf(docFreq=108, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
         0.11622233 = weight(abstract_txt:runs in 910) [ClassicSimilarity], result of:
           0.11622233 = score(doc=910,freq=1.0), product of:
             0.18906957 = queryWeight, product of:
               1.1692859 = boost
               7.8682456 = idf(docFreq=45, maxDocs=44218)
               0.020550529 = queryNorm
             0.6147067 = fieldWeight in 910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.8682456 = idf(docFreq=45, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
         0.021990389 = weight(abstract_txt:that in 910) [ClassicSimilarity], result of:
           0.021990389 = score(doc=910,freq=3.0), product of:
             0.06858515 = queryWeight, product of:
               1.4084939 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020550529 = queryNorm
             0.320629 = fieldWeight in 910, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
         0.23301092 = weight(abstract_txt:metrics in 910) [ClassicSimilarity], result of:
           0.23301092 = score(doc=910,freq=5.0), product of:
             0.20124257 = queryWeight, product of:
               1.4774591 = boost
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.020550529 = queryNorm
             1.157861 = fieldWeight in 910, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               6.627983 = idf(docFreq=158, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
         0.14007649 = weight(abstract_txt:metric in 910) [ClassicSimilarity], result of:
           0.14007649 = score(doc=910,freq=1.0), product of:
             0.2451146 = queryWeight, product of:
               1.6305726 = boost
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.020550529 = queryNorm
             0.5714735 = fieldWeight in 910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.314861 = idf(docFreq=79, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
         0.06605226 = weight(abstract_txt:document in 910) [ClassicSimilarity], result of:
           0.06605226 = score(doc=910,freq=1.0), product of:
             0.19695924 = queryWeight, product of:
               2.232709 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.020550529 = queryNorm
             0.33536002 = fieldWeight in 910, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=910)
       0.24 = coord(6/25)
   

2. Lai, C,-H.: Applying knowledge flow mining to group recommendation methods for task-based groups (2015) 0.16

   0.15573932 = sum of:
     0.15573932 = product of:
       0.5562119 = sum of:
         0.040198818 = weight(abstract_txt:performance in 1667) [ClassicSimilarity], result of:
           0.040198818 = score(doc=1667,freq=2.0), product of:
             0.098219536 = queryWeight, product of:
               1.0321777 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.020550529 = queryNorm
             0.40927517 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.08308107 = weight(abstract_txt:task in 1667) [ClassicSimilarity], result of:
           0.08308107 = score(doc=1667,freq=6.0), product of:
             0.11049686 = queryWeight, product of:
               1.0947893 = boost
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.020550529 = queryNorm
             0.7518862 = fieldWeight in 1667, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.038539946 = weight(abstract_txt:time in 1667) [ClassicSimilarity], result of:
           0.038539946 = score(doc=1667,freq=2.0), product of:
             0.10510967 = queryWeight, product of:
               1.2329522 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.020550529 = queryNorm
             0.36666414 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.041282482 = weight(abstract_txt:over in 1667) [ClassicSimilarity], result of:
           0.041282482 = score(doc=1667,freq=2.0), product of:
             0.110038795 = queryWeight, product of:
               1.2615306 = boost
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.020550529 = queryNorm
             0.375163 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.244485 = idf(docFreq=1723, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.014364062 = weight(abstract_txt:that in 1667) [ClassicSimilarity], result of:
           0.014364062 = score(doc=1667,freq=2.0), product of:
             0.06858515 = queryWeight, product of:
               1.4084939 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020550529 = queryNorm
             0.20943399 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.0747296 = weight(abstract_txt:document in 1667) [ClassicSimilarity], result of:
           0.0747296 = score(doc=1667,freq=2.0), product of:
             0.19695924 = queryWeight, product of:
               2.232709 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.020550529 = queryNorm
             0.37941656 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
         0.2640159 = weight(abstract_txt:filtering in 1667) [ClassicSimilarity], result of:
           0.2640159 = score(doc=1667,freq=2.0), product of:
             0.45687917 = queryWeight, product of:
               3.4005144 = boost
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.020550529 = queryNorm
             0.57786816 = fieldWeight in 1667, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.0625 = fieldNorm(doc=1667)
       0.28 = coord(7/25)
   

3. Ku, C.-H.; Leroy, G.: ¬A crime reports analysis system to identify related crimes (2011) 0.14

   0.14237471 = sum of:
     0.14237471 = product of:
       0.444921 = sum of:
         0.02584847 = weight(abstract_txt:evaluation in 4629) [ClassicSimilarity], result of:
           0.02584847 = score(doc=4629,freq=1.0), product of:
             0.092191085 = queryWeight, product of:
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.020550529 = queryNorm
             0.2803793 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.028424857 = weight(abstract_txt:performance in 4629) [ClassicSimilarity], result of:
           0.028424857 = score(doc=4629,freq=1.0), product of:
             0.098219536 = queryWeight, product of:
               1.0321777 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.020550529 = queryNorm
             0.28940126 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.033917703 = weight(abstract_txt:task in 4629) [ClassicSimilarity], result of:
           0.033917703 = score(doc=4629,freq=1.0), product of:
             0.11049686 = queryWeight, product of:
               1.0947893 = boost
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.020550529 = queryNorm
             0.30695623 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.9112997 = idf(docFreq=884, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.027251856 = weight(abstract_txt:time in 4629) [ClassicSimilarity], result of:
           0.027251856 = score(doc=4629,freq=1.0), product of:
             0.10510967 = queryWeight, product of:
               1.2329522 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.020550529 = queryNorm
             0.2592707 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.014364062 = weight(abstract_txt:that in 4629) [ClassicSimilarity], result of:
           0.014364062 = score(doc=4629,freq=2.0), product of:
             0.06858515 = queryWeight, product of:
               1.4084939 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020550529 = queryNorm
             0.20943399 = fieldWeight in 4629, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.022742998 = weight(abstract_txt:systems in 4629) [ClassicSimilarity], result of:
           0.022742998 = score(doc=4629,freq=1.0), product of:
             0.106653266 = queryWeight, product of:
               1.5210994 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.020550529 = queryNorm
             0.2132424 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.10568361 = weight(abstract_txt:document in 4629) [ClassicSimilarity], result of:
           0.10568361 = score(doc=4629,freq=4.0), product of:
             0.19695924 = queryWeight, product of:
               2.232709 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.020550529 = queryNorm
             0.53657603 = fieldWeight in 4629, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
         0.18668744 = weight(abstract_txt:filtering in 4629) [ClassicSimilarity], result of:
           0.18668744 = score(doc=4629,freq=1.0), product of:
             0.45687917 = queryWeight, product of:
               3.4005144 = boost
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.020550529 = queryNorm
             0.4086145 = fieldWeight in 4629, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.0625 = fieldNorm(doc=4629)
       0.32 = coord(8/25)
   

4. Pera, M.S.; Ng, Y.-K.: SpamED : a spam E-mail detection approach based on phrase similarity (2009) 0.13

   0.12980635 = sum of:
     0.12980635 = product of:
       0.64903176 = sum of:
         0.15026033 = weight(abstract_txt:false in 2721) [ClassicSimilarity], result of:
           0.15026033 = score(doc=2721,freq=2.0), product of:
             0.17809354 = queryWeight, product of:
               1.1348383 = boost
               7.636444 = idf(docFreq=57, maxDocs=44218)
               0.020550529 = queryNorm
             0.8437158 = fieldWeight in 2721, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.636444 = idf(docFreq=57, maxDocs=44218)
               0.078125 = fieldNorm(doc=2721)
         0.048174933 = weight(abstract_txt:time in 2721) [ClassicSimilarity], result of:
           0.048174933 = score(doc=2721,freq=2.0), product of:
             0.10510967 = queryWeight, product of:
               1.2329522 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.020550529 = queryNorm
             0.45833018 = fieldWeight in 2721, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.078125 = fieldNorm(doc=2721)
         0.012696157 = weight(abstract_txt:that in 2721) [ClassicSimilarity], result of:
           0.012696157 = score(doc=2721,freq=1.0), product of:
             0.06858515 = queryWeight, product of:
               1.4084939 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020550529 = queryNorm
             0.18511525 = fieldWeight in 2721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=2721)
         0.204541 = weight(abstract_txt:positives in 2721) [ClassicSimilarity], result of:
           0.204541 = score(doc=2721,freq=1.0), product of:
             0.27560186 = queryWeight, product of:
               1.4117279 = boost
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.020550529 = queryNorm
             0.74216115 = fieldWeight in 2721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.499662 = idf(docFreq=8, maxDocs=44218)
               0.078125 = fieldNorm(doc=2721)
         0.2333593 = weight(abstract_txt:filtering in 2721) [ClassicSimilarity], result of:
           0.2333593 = score(doc=2721,freq=1.0), product of:
             0.45687917 = queryWeight, product of:
               3.4005144 = boost
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.020550529 = queryNorm
             0.5107681 = fieldWeight in 2721, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.078125 = fieldNorm(doc=2721)
       0.2 = coord(5/25)
   

5. Toepfer, M.; Seifert, C.: Content-based quality estimation for automatic subject indexing of short texts under precision and recall constraints 0.13

   0.12682903 = sum of:
     0.12682903 = product of:
       0.5284543 = sum of:
         0.11719902 = weight(abstract_txt:estimation in 4309) [ClassicSimilarity], result of:
           0.11719902 = score(doc=4309,freq=1.0), product of:
             0.19012733 = queryWeight, product of:
               1.1725522 = boost
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.020550529 = queryNorm
             0.61642385 = fieldWeight in 4309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.890225 = idf(docFreq=44, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
         0.034064822 = weight(abstract_txt:time in 4309) [ClassicSimilarity], result of:
           0.034064822 = score(doc=4309,freq=1.0), product of:
             0.10510967 = queryWeight, product of:
               1.2329522 = boost
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.020550529 = queryNorm
             0.32408836 = fieldWeight in 4309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.148331 = idf(docFreq=1897, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
         0.021990389 = weight(abstract_txt:that in 4309) [ClassicSimilarity], result of:
           0.021990389 = score(doc=4309,freq=3.0), product of:
             0.06858515 = queryWeight, product of:
               1.4084939 = boost
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.020550529 = queryNorm
             0.320629 = fieldWeight in 4309, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               2.3694751 = idf(docFreq=11241, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
         0.028428746 = weight(abstract_txt:systems in 4309) [ClassicSimilarity], result of:
           0.028428746 = score(doc=4309,freq=1.0), product of:
             0.106653266 = queryWeight, product of:
               1.5210994 = boost
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.020550529 = queryNorm
             0.26655298 = fieldWeight in 4309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.4118783 = idf(docFreq=3963, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
         0.093412 = weight(abstract_txt:document in 4309) [ClassicSimilarity], result of:
           0.093412 = score(doc=4309,freq=2.0), product of:
             0.19695924 = queryWeight, product of:
               2.232709 = boost
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.020550529 = queryNorm
             0.4742707 = fieldWeight in 4309, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.2926083 = idf(docFreq=1642, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
         0.2333593 = weight(abstract_txt:filtering in 4309) [ClassicSimilarity], result of:
           0.2333593 = score(doc=4309,freq=1.0), product of:
             0.45687917 = queryWeight, product of:
               3.4005144 = boost
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.020550529 = queryNorm
             0.5107681 = fieldWeight in 4309, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.537832 = idf(docFreq=173, maxDocs=44218)
               0.078125 = fieldNorm(doc=4309)
       0.24 = coord(6/25)