Document (#6934)

Author

Su, L.T.

Title

¬The relevance of recall and precision in user evaluation

Source

Journal of the American Society for Information Science. 45(1994) no.3, S.207-217

Year

1994

Abstract

The appropriateness of evaluation criteria and measures have been a subject of debate and a vital concern in the information retrieval evaluation literature. A study was conducted to investigate the appropriateness of 20 measures for evaluating interactive information retrieval performance, representing 4 major evaluation criteria. Among the 20 measures studied were the 2 most well-known relevance-based measures of effectiveness, recall and precision. The user's judgment of information success was used as the devised criterion measure with which all other 20 measures were to be correlated. A sample of 40 end-users with individual information problems from an academic environment were observed, interacting with 6 professional intermediaries searching on their behalf in large operational systems. Quantitative data consisting of values for all measures studies and verbal data containing users' reasons for assigning certain values to selected measures were collected. Statistical analysis of the quantitative data showed that precision, one of the most important traditional measures of effectiveness, is not sifnificantly correlated with the user's judgment of success. Users appear to be more concerned with absolute recall than with precision, although absolute recall was not directly tested in this study. 4 related measures of recall and precision are found to be significantly correlated with success. Among these are user's satisfaction with completeness of search results and user's satisfaction with precision of the search. This article explores the possible explanations for this outcome through content analysis of users' verbal data. The analysis shows that high precision does not always mean high quality (relevancy, completeness, etc.) to users because of different users' expectations. The user's purpose in obtaining information is suggested to be the primary cause for the high concern for recall. Implications for research and practice are discussed

Similar documents (content)

1. Su, L.T.: ¬A comprehensive and systematic model of user evaluation of Web search engines : Il. An evaluation by undergraduates (2003) 0.51

   0.5139616 = sum of:
     0.5139616 = product of:
       0.9177886 = sum of:
         0.023720425 = weight(abstract_txt:analysis in 2117) [ClassicSimilarity], result of:
           0.023720425 = score(doc=2117,freq=3.0), product of:
             0.0685424 = queryWeight, product of:
               1.0749139 = boost
               3.6535451 = idf(docFreq=3112, maxDocs=44218)
               0.017453043 = queryNorm
             0.34606937 = fieldWeight in 2117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.6535451 = idf(docFreq=3112, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.056166947 = weight(abstract_txt:criteria in 2117) [ClassicSimilarity], result of:
           0.056166947 = score(doc=2117,freq=3.0), product of:
             0.10637374 = queryWeight, product of:
               1.0933661 = boost
               5.574394 = idf(docFreq=455, maxDocs=44218)
               0.017453043 = queryNorm
             0.52801514 = fieldWeight in 2117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.574394 = idf(docFreq=455, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.03976938 = weight(abstract_txt:quantitative in 2117) [ClassicSimilarity], result of:
           0.03976938 = score(doc=2117,freq=1.0), product of:
             0.121876575 = queryWeight, product of:
               1.1703305 = boost
               5.9667873 = idf(docFreq=307, maxDocs=44218)
               0.017453043 = queryNorm
             0.32630867 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.9667873 = idf(docFreq=307, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.009391559 = weight(abstract_txt:information in 2117) [ClassicSimilarity], result of:
           0.009391559 = score(doc=2117,freq=2.0), product of:
             0.050159067 = queryWeight, product of:
               1.1871159 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017453043 = queryNorm
             0.18723552 = fieldWeight in 2117, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.120696984 = weight(abstract_txt:satisfaction in 2117) [ClassicSimilarity], result of:
           0.120696984 = score(doc=2117,freq=6.0), product of:
             0.14059445 = queryWeight, product of:
               1.256992 = boost
               6.4086204 = idf(docFreq=197, maxDocs=44218)
               0.017453043 = queryNorm
             0.85847616 = fieldWeight in 2117, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               6.4086204 = idf(docFreq=197, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.013905006 = weight(abstract_txt:data in 2117) [ClassicSimilarity], result of:
           0.013905006 = score(doc=2117,freq=1.0), product of:
             0.07620997 = queryWeight, product of:
               1.308788 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.017453043 = queryNorm
             0.18245652 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.0646197 = weight(abstract_txt:verbal in 2117) [ClassicSimilarity], result of:
           0.0646197 = score(doc=2117,freq=1.0), product of:
             0.16844738 = queryWeight, product of:
               1.3758799 = boost
               7.014756 = idf(docFreq=107, maxDocs=44218)
               0.017453043 = queryNorm
             0.3836195 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.014756 = idf(docFreq=107, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.018508788 = weight(abstract_txt:were in 2117) [ClassicSimilarity], result of:
           0.018508788 = score(doc=2117,freq=1.0), product of:
             0.09221812 = queryWeight, product of:
               1.4396985 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.017453043 = queryNorm
             0.20070662 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.06760595 = weight(abstract_txt:evaluation in 2117) [ClassicSimilarity], result of:
           0.06760595 = score(doc=2117,freq=4.0), product of:
             0.13778467 = queryWeight, product of:
               1.7598022 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017453043 = queryNorm
             0.49066377 = fieldWeight in 2117, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.02554889 = weight(abstract_txt:users in 2117) [ClassicSimilarity], result of:
           0.02554889 = score(doc=2117,freq=1.0), product of:
             0.1308708 = queryWeight, product of:
               2.1005373 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.017453043 = queryNorm
             0.19522223 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.026317874 = weight(abstract_txt:with in 2117) [ClassicSimilarity], result of:
           0.026317874 = score(doc=2117,freq=4.0), product of:
             0.096258484 = queryWeight, product of:
               2.2063487 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.017453043 = queryNorm
             0.27340835 = fieldWeight in 2117, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.106706746 = weight(abstract_txt:recall in 2117) [ClassicSimilarity], result of:
           0.106706746 = score(doc=2117,freq=1.0), product of:
             0.33940846 = queryWeight, product of:
               3.3827536 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.017453043 = queryNorm
             0.31439036 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.11051878 = weight(abstract_txt:precision in 2117) [ClassicSimilarity], result of:
           0.11051878 = score(doc=2117,freq=1.0), product of:
             0.36576408 = queryWeight, product of:
               3.7930012 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.017453043 = queryNorm
             0.30215865 = fieldWeight in 2117, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
         0.2343116 = weight(abstract_txt:measures in 2117) [ClassicSimilarity], result of:
           0.2343116 = score(doc=2117,freq=3.0), product of:
             0.45510745 = queryWeight, product of:
               4.797463 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017453043 = queryNorm
             0.51484895 = fieldWeight in 2117, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.0546875 = fieldNorm(doc=2117)
       0.56 = coord(14/25)
   

2. Lesk, M.E.; Salton, G.: Relevance assements and retrieval system evaluation (1969) 0.38

   0.37938872 = sum of:
     0.37938872 = product of:
       1.1855898 = sum of:
         0.06014804 = weight(abstract_txt:effectiveness in 4151) [ClassicSimilarity], result of:
           0.06014804 = score(doc=4151,freq=2.0), product of:
             0.08898221 = queryWeight, product of:
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.017453043 = queryNorm
             0.67595583 = fieldWeight in 4151, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.05559086 = weight(abstract_txt:criteria in 4151) [ClassicSimilarity], result of:
           0.05559086 = score(doc=4151,freq=1.0), product of:
             0.10637374 = queryWeight, product of:
               1.0933661 = boost
               5.574394 = idf(docFreq=455, maxDocs=44218)
               0.017453043 = queryNorm
             0.52259946 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.574394 = idf(docFreq=455, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.011384289 = weight(abstract_txt:information in 4151) [ClassicSimilarity], result of:
           0.011384289 = score(doc=4151,freq=1.0), product of:
             0.050159067 = queryWeight, product of:
               1.1871159 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017453043 = queryNorm
             0.22696373 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.023837153 = weight(abstract_txt:data in 4151) [ClassicSimilarity], result of:
           0.023837153 = score(doc=4151,freq=1.0), product of:
             0.07620997 = queryWeight, product of:
               1.308788 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.017453043 = queryNorm
             0.31278262 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.057947956 = weight(abstract_txt:evaluation in 4151) [ClassicSimilarity], result of:
           0.057947956 = score(doc=4151,freq=1.0), product of:
             0.13778467 = queryWeight, product of:
               1.7598022 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017453043 = queryNorm
             0.42056894 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.3658517 = weight(abstract_txt:recall in 4151) [ClassicSimilarity], result of:
           0.3658517 = score(doc=4151,freq=4.0), product of:
             0.33940846 = queryWeight, product of:
               3.3827536 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.017453043 = queryNorm
             1.0779098 = fieldWeight in 4151, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.3789215 = weight(abstract_txt:precision in 4151) [ClassicSimilarity], result of:
           0.3789215 = score(doc=4151,freq=4.0), product of:
             0.36576408 = queryWeight, product of:
               3.7930012 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.017453043 = queryNorm
             1.0359725 = fieldWeight in 4151, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
         0.23190835 = weight(abstract_txt:measures in 4151) [ClassicSimilarity], result of:
           0.23190835 = score(doc=4151,freq=1.0), product of:
             0.45510745 = queryWeight, product of:
               4.797463 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017453043 = queryNorm
             0.50956833 = fieldWeight in 4151, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.09375 = fieldNorm(doc=4151)
       0.32 = coord(8/25)
   

3. Wildemuth, B.M.: Measures of success in searching a full-text fact base (1990) 0.33

   0.32818344 = sum of:
     0.32818344 = product of:
       1.1720837 = sum of:
         0.041287158 = weight(abstract_txt:data in 2050) [ClassicSimilarity], result of:
           0.041287158 = score(doc=2050,freq=3.0), product of:
             0.07620997 = queryWeight, product of:
               1.308788 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.017453043 = queryNorm
             0.5417553 = fieldWeight in 2050, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.031729348 = weight(abstract_txt:were in 2050) [ClassicSimilarity], result of:
           0.031729348 = score(doc=2050,freq=1.0), product of:
             0.09221812 = queryWeight, product of:
               1.4396985 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.017453043 = queryNorm
             0.34406847 = fieldWeight in 2050, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.12643874 = weight(abstract_txt:success in 2050) [ClassicSimilarity], result of:
           0.12643874 = score(doc=2050,freq=2.0), product of:
             0.16714987 = queryWeight, product of:
               1.6785994 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.017453043 = queryNorm
             0.7564393 = fieldWeight in 2050, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.057947956 = weight(abstract_txt:evaluation in 2050) [ClassicSimilarity], result of:
           0.057947956 = score(doc=2050,freq=1.0), product of:
             0.13778467 = queryWeight, product of:
               1.7598022 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017453043 = queryNorm
             0.42056894 = fieldWeight in 2050, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.18292585 = weight(abstract_txt:recall in 2050) [ClassicSimilarity], result of:
           0.18292585 = score(doc=2050,freq=1.0), product of:
             0.33940846 = queryWeight, product of:
               3.3827536 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.017453043 = queryNorm
             0.5389549 = fieldWeight in 2050, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.267938 = weight(abstract_txt:precision in 2050) [ClassicSimilarity], result of:
           0.267938 = score(doc=2050,freq=2.0), product of:
             0.36576408 = queryWeight, product of:
               3.7930012 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.017453043 = queryNorm
             0.7325432 = fieldWeight in 2050, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
         0.4638167 = weight(abstract_txt:measures in 2050) [ClassicSimilarity], result of:
           0.4638167 = score(doc=2050,freq=4.0), product of:
             0.45510745 = queryWeight, product of:
               4.797463 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017453043 = queryNorm
             1.0191367 = fieldWeight in 2050, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.09375 = fieldNorm(doc=2050)
       0.28 = coord(7/25)
   

4. Huuskonen, S.; Vakkari, P.: Students' search process and outcome in Medline in writing an essay for a class on evidence-based medicine (2008) 0.32

   0.3243059 = sum of:
     0.3243059 = product of:
       0.90084964 = sum of:
         0.016266657 = weight(abstract_txt:information in 1891) [ClassicSimilarity], result of:
           0.016266657 = score(doc=1891,freq=6.0), product of:
             0.050159067 = queryWeight, product of:
               1.1871159 = boost
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.017453043 = queryNorm
             0.32430142 = fieldWeight in 1891, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               2.4209464 = idf(docFreq=10677, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.013905006 = weight(abstract_txt:data in 1891) [ClassicSimilarity], result of:
           0.013905006 = score(doc=1891,freq=1.0), product of:
             0.07620997 = queryWeight, product of:
               1.308788 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.017453043 = queryNorm
             0.18245652 = fieldWeight in 1891, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.037017576 = weight(abstract_txt:were in 1891) [ClassicSimilarity], result of:
           0.037017576 = score(doc=1891,freq=4.0), product of:
             0.09221812 = queryWeight, product of:
               1.4396985 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.017453043 = queryNorm
             0.40141323 = fieldWeight in 1891, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.047804624 = weight(abstract_txt:evaluation in 1891) [ClassicSimilarity], result of:
           0.047804624 = score(doc=1891,freq=2.0), product of:
             0.13778467 = queryWeight, product of:
               1.7598022 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017453043 = queryNorm
             0.34695166 = fieldWeight in 1891, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.03613159 = weight(abstract_txt:users in 1891) [ClassicSimilarity], result of:
           0.03613159 = score(doc=1891,freq=2.0), product of:
             0.1308708 = queryWeight, product of:
               2.1005373 = boost
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.017453043 = queryNorm
             0.27608594 = fieldWeight in 1891, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.569778 = idf(docFreq=3384, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.026317874 = weight(abstract_txt:with in 1891) [ClassicSimilarity], result of:
           0.026317874 = score(doc=1891,freq=4.0), product of:
             0.096258484 = queryWeight, product of:
               2.2063487 = boost
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.017453043 = queryNorm
             0.27340835 = fieldWeight in 1891, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               2.4997334 = idf(docFreq=9868, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.26137707 = weight(abstract_txt:recall in 1891) [ClassicSimilarity], result of:
           0.26137707 = score(doc=1891,freq=6.0), product of:
             0.33940846 = queryWeight, product of:
               3.3827536 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.017453043 = queryNorm
             0.77009594 = fieldWeight in 1891, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.2707146 = weight(abstract_txt:precision in 1891) [ClassicSimilarity], result of:
           0.2707146 = score(doc=1891,freq=6.0), product of:
             0.36576408 = queryWeight, product of:
               3.7930012 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.017453043 = queryNorm
             0.7401345 = fieldWeight in 1891, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
         0.19131462 = weight(abstract_txt:measures in 1891) [ClassicSimilarity], result of:
           0.19131462 = score(doc=1891,freq=2.0), product of:
             0.45510745 = queryWeight, product of:
               4.797463 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017453043 = queryNorm
             0.42037243 = fieldWeight in 1891, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.0546875 = fieldNorm(doc=1891)
       0.36 = coord(9/25)
   

5. Rowley, J.E.: CD-ROM versus online : an evaluation of the effects of the user interface on search effectiveness; a pilot study (1989) 0.32

   0.31539604 = sum of:
     0.31539604 = product of:
       0.98561263 = sum of:
         0.049619608 = weight(abstract_txt:effectiveness in 2873) [ClassicSimilarity], result of:
           0.049619608 = score(doc=2873,freq=1.0), product of:
             0.08898221 = queryWeight, product of:
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.017453043 = queryNorm
             0.5576351 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.098378 = idf(docFreq=733, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.09854868 = weight(abstract_txt:satisfaction in 2873) [ClassicSimilarity], result of:
           0.09854868 = score(doc=2873,freq=1.0), product of:
             0.14059445 = queryWeight, product of:
               1.256992 = boost
               6.4086204 = idf(docFreq=197, maxDocs=44218)
               0.017453043 = queryNorm
             0.7009429 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4086204 = idf(docFreq=197, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.027810013 = weight(abstract_txt:data in 2873) [ClassicSimilarity], result of:
           0.027810013 = score(doc=2873,freq=1.0), product of:
             0.07620997 = queryWeight, product of:
               1.308788 = boost
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.017453043 = queryNorm
             0.36491305 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.3363478 = idf(docFreq=4274, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.037017576 = weight(abstract_txt:were in 2873) [ClassicSimilarity], result of:
           0.037017576 = score(doc=2873,freq=1.0), product of:
             0.09221812 = queryWeight, product of:
               1.4396985 = boost
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.017453043 = queryNorm
             0.40141323 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.6700637 = idf(docFreq=3061, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.06760595 = weight(abstract_txt:evaluation in 2873) [ClassicSimilarity], result of:
           0.06760595 = score(doc=2873,freq=1.0), product of:
             0.13778467 = queryWeight, product of:
               1.7598022 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.017453043 = queryNorm
             0.49066377 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.21341349 = weight(abstract_txt:recall in 2873) [ClassicSimilarity], result of:
           0.21341349 = score(doc=2873,freq=1.0), product of:
             0.33940846 = queryWeight, product of:
               3.3827536 = boost
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.017453043 = queryNorm
             0.6287807 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.7488523 = idf(docFreq=382, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.22103757 = weight(abstract_txt:precision in 2873) [ClassicSimilarity], result of:
           0.22103757 = score(doc=2873,freq=1.0), product of:
             0.36576408 = queryWeight, product of:
               3.7930012 = boost
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.017453043 = queryNorm
             0.6043173 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.5251865 = idf(docFreq=478, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
         0.27055976 = weight(abstract_txt:measures in 2873) [ClassicSimilarity], result of:
           0.27055976 = score(doc=2873,freq=1.0), product of:
             0.45510745 = queryWeight, product of:
               4.797463 = boost
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.017453043 = queryNorm
             0.5944964 = fieldWeight in 2873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.4353957 = idf(docFreq=523, maxDocs=44218)
               0.109375 = fieldNorm(doc=2873)
       0.32 = coord(8/25)