Document (#39682)

Author

Abdi, A.
Idris, N.
Alguliev, R.M.
Aliguliyev, R.M.

Title

Automatic summarization assessment through a combination of semantic and syntactic information for intelligent educational systems

Source

Information processing and management. 51(2015) no.4, S.340-358

Year

2015

Abstract

Summary writing is a process for creating a short version of a source text. It can be used as a measure of understanding. As grading students' summaries is a very time-consuming task, computer-assisted assessment can help teachers perform the grading more effectively. Several techniques, such as BLEU, ROUGE, N-gram co-occurrence, Latent Semantic Analysis (LSA), LSA_Ngram and LSA_ERB, have been proposed to support the automatic assessment of students' summaries. Since these techniques are more suitable for long texts, their performance is not satisfactory for the evaluation of short summaries. This paper proposes a specialized method that works well in assessing short summaries. Our proposed method integrates the semantic relations between words, and their syntactic composition. As a result, the proposed method is able to obtain high accuracy and improve the performance compared with the current techniques. Experiments have displayed that it is to be preferred over the existing techniques. A summary evaluation system based on the proposed method has also been developed.

Content

Vgl.: doi: 10.1016/j.ipm.2015.02.001.

Theme

Automatisches Abstracting

Similar documents (content)

1. Sjöbergh, J.: Older versions of the ROUGEeval summarization evaluation system were easier to fool (2007) 0.30

   0.2974638 = sum of:
     0.2974638 = product of:
       1.2394326 = sum of:
         0.13414311 = weight(abstract_txt:summarization in 940) [ClassicSimilarity], result of:
           0.13414311 = score(doc=940,freq=2.0), product of:
             0.121588126 = queryWeight, product of:
               1.0154442 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.016787691 = queryNorm
             1.1032584 = fieldWeight in 940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
         0.04720073 = weight(abstract_txt:evaluation in 940) [ClassicSimilarity], result of:
           0.04720073 = score(doc=940,freq=1.0), product of:
             0.0961977 = queryWeight, product of:
               1.2773439 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016787691 = queryNorm
             0.49066377 = fieldWeight in 940, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
         0.27473143 = weight(abstract_txt:rouge in 940) [ClassicSimilarity], result of:
           0.27473143 = score(doc=940,freq=2.0), product of:
             0.19608806 = queryWeight, product of:
               1.2895436 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.016787691 = queryNorm
             1.4010615 = fieldWeight in 940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
         0.103698485 = weight(abstract_txt:automatic in 940) [ClassicSimilarity], result of:
           0.103698485 = score(doc=940,freq=2.0), product of:
             0.12903391 = queryWeight, product of:
               1.479372 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.016787691 = queryNorm
             0.803653 = fieldWeight in 940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
         0.16514067 = weight(abstract_txt:method in 940) [ClassicSimilarity], result of:
           0.16514067 = score(doc=940,freq=3.0), product of:
             0.19367395 = queryWeight, product of:
               2.563162 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.016787691 = queryNorm
             0.85267365 = fieldWeight in 940, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
         0.5145182 = weight(abstract_txt:summaries in 940) [ClassicSimilarity], result of:
           0.5145182 = score(doc=940,freq=2.0), product of:
             0.4729328 = queryWeight, product of:
               4.0053473 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.016787691 = queryNorm
             1.0879309 = fieldWeight in 940, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.109375 = fieldNorm(doc=940)
       0.24 = coord(6/25)
   

2. Ou, S.; Khoo, S.G.; Goh, D.H.: Automatic multidocument summarization of research abstracts : design and user evaluation (2007) 0.29

   0.28532073 = sum of:
     0.28532073 = product of:
       0.89162725 = sum of:
         0.05176631 = weight(abstract_txt:integrates in 522) [ClassicSimilarity], result of:
           0.05176631 = score(doc=522,freq=1.0), product of:
             0.117917724 = queryWeight, product of:
               7.0240583 = idf(docFreq=106, maxDocs=44218)
               0.016787691 = queryNorm
             0.43900365 = fieldWeight in 522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.0240583 = idf(docFreq=106, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.076653205 = weight(abstract_txt:summarization in 522) [ClassicSimilarity], result of:
           0.076653205 = score(doc=522,freq=2.0), product of:
             0.121588126 = queryWeight, product of:
               1.0154442 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.016787691 = queryNorm
             0.6304333 = fieldWeight in 522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.038143948 = weight(abstract_txt:evaluation in 522) [ClassicSimilarity], result of:
           0.038143948 = score(doc=522,freq=2.0), product of:
             0.0961977 = queryWeight, product of:
               1.2773439 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016787691 = queryNorm
             0.3965162 = fieldWeight in 522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.041900516 = weight(abstract_txt:automatic in 522) [ClassicSimilarity], result of:
           0.041900516 = score(doc=522,freq=1.0), product of:
             0.12903391 = queryWeight, product of:
               1.479372 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.016787691 = queryNorm
             0.32472485 = fieldWeight in 522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.11449073 = weight(abstract_txt:summary in 522) [ClassicSimilarity], result of:
           0.11449073 = score(doc=522,freq=2.0), product of:
             0.20016794 = queryWeight, product of:
               1.8425646 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.016787691 = queryNorm
             0.5719734 = fieldWeight in 522, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.0943661 = weight(abstract_txt:method in 522) [ClassicSimilarity], result of:
           0.0943661 = score(doc=522,freq=3.0), product of:
             0.19367395 = queryWeight, product of:
               2.563162 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.016787691 = queryNorm
             0.4872421 = fieldWeight in 522, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.058512896 = weight(abstract_txt:proposed in 522) [ClassicSimilarity], result of:
           0.058512896 = score(doc=522,freq=1.0), product of:
             0.20311189 = queryWeight, product of:
               2.624872 = boost
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.016787691 = queryNorm
             0.2880821 = fieldWeight in 522, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
         0.41579354 = weight(abstract_txt:summaries in 522) [ClassicSimilarity], result of:
           0.41579354 = score(doc=522,freq=4.0), product of:
             0.4729328 = queryWeight, product of:
               4.0053473 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.016787691 = queryNorm
             0.879181 = fieldWeight in 522, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.0625 = fieldNorm(doc=522)
       0.32 = coord(8/25)
   

3. Hirao, T.; Okumura, M.; Yasuda, N.; Isozaki, H.: Supervised automatic evaluation for summarization with voted regression model (2007) 0.26

   0.2559227 = sum of:
     0.2559227 = product of:
       0.91400963 = sum of:
         0.09581651 = weight(abstract_txt:summarization in 942) [ClassicSimilarity], result of:
           0.09581651 = score(doc=942,freq=2.0), product of:
             0.121588126 = queryWeight, product of:
               1.0154442 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.016787691 = queryNorm
             0.78804165 = fieldWeight in 942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.017679784 = weight(abstract_txt:been in 942) [ClassicSimilarity], result of:
           0.017679784 = score(doc=942,freq=1.0), product of:
             0.06255599 = queryWeight, product of:
               1.0300539 = boost
               3.617579 = idf(docFreq=3226, maxDocs=44218)
               0.016787691 = queryNorm
             0.28262335 = fieldWeight in 942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.617579 = idf(docFreq=3226, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.09535987 = weight(abstract_txt:evaluation in 942) [ClassicSimilarity], result of:
           0.09535987 = score(doc=942,freq=8.0), product of:
             0.0961977 = queryWeight, product of:
               1.2773439 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016787691 = queryNorm
             0.9912905 = fieldWeight in 942, product of:
               2.828427 = tf(freq=8.0), with freq of:
                 8.0 = termFreq=8.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.1282936 = weight(abstract_txt:automatic in 942) [ClassicSimilarity], result of:
           0.1282936 = score(doc=942,freq=6.0), product of:
             0.12903391 = queryWeight, product of:
               1.479372 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.016787691 = queryNorm
             0.99426275 = fieldWeight in 942, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.13620575 = weight(abstract_txt:method in 942) [ClassicSimilarity], result of:
           0.13620575 = score(doc=942,freq=4.0), product of:
             0.19367395 = queryWeight, product of:
               2.563162 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.016787691 = queryNorm
             0.7032734 = fieldWeight in 942, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.07314112 = weight(abstract_txt:proposed in 942) [ClassicSimilarity], result of:
           0.07314112 = score(doc=942,freq=1.0), product of:
             0.20311189 = queryWeight, product of:
               2.624872 = boost
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.016787691 = queryNorm
             0.36010262 = fieldWeight in 942, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
         0.367513 = weight(abstract_txt:summaries in 942) [ClassicSimilarity], result of:
           0.367513 = score(doc=942,freq=2.0), product of:
             0.4729328 = queryWeight, product of:
               4.0053473 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.016787691 = queryNorm
             0.7770935 = fieldWeight in 942, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.078125 = fieldNorm(doc=942)
       0.28 = coord(7/25)
   

4. Ou, S.; Khoo, C.S.G.; Goh, D.H.: Multi-document summarization of news articles using an event-based framework (2006) 0.25

   0.2475558 = sum of:
     0.2475558 = product of:
       0.88412786 = sum of:
         0.076653205 = weight(abstract_txt:summarization in 657) [ClassicSimilarity], result of:
           0.076653205 = score(doc=657,freq=2.0), product of:
             0.121588126 = queryWeight, product of:
               1.0154442 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.016787691 = queryNorm
             0.6304333 = fieldWeight in 657, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.038143948 = weight(abstract_txt:evaluation in 657) [ClassicSimilarity], result of:
           0.038143948 = score(doc=657,freq=2.0), product of:
             0.0961977 = queryWeight, product of:
               1.2773439 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016787691 = queryNorm
             0.3965162 = fieldWeight in 657, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.041900516 = weight(abstract_txt:automatic in 657) [ClassicSimilarity], result of:
           0.041900516 = score(doc=657,freq=1.0), product of:
             0.12903391 = queryWeight, product of:
               1.479372 = boost
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.016787691 = queryNorm
             0.32472485 = fieldWeight in 657, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.1955976 = idf(docFreq=665, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.080957174 = weight(abstract_txt:summary in 657) [ClassicSimilarity], result of:
           0.080957174 = score(doc=657,freq=1.0), product of:
             0.20016794 = queryWeight, product of:
               1.8425646 = boost
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.016787691 = queryNorm
             0.40444627 = fieldWeight in 657, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.4711404 = idf(docFreq=185, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.054482296 = weight(abstract_txt:method in 657) [ClassicSimilarity], result of:
           0.054482296 = score(doc=657,freq=1.0), product of:
             0.19367395 = queryWeight, product of:
               2.563162 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.016787691 = queryNorm
             0.28130937 = fieldWeight in 657, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.08274973 = weight(abstract_txt:proposed in 657) [ClassicSimilarity], result of:
           0.08274973 = score(doc=657,freq=2.0), product of:
             0.20311189 = queryWeight, product of:
               2.624872 = boost
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.016787691 = queryNorm
             0.4074096 = fieldWeight in 657, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
         0.509241 = weight(abstract_txt:summaries in 657) [ClassicSimilarity], result of:
           0.509241 = score(doc=657,freq=6.0), product of:
             0.4729328 = queryWeight, product of:
               4.0053473 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.016787691 = queryNorm
             1.0767725 = fieldWeight in 657, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.0625 = fieldNorm(doc=657)
       0.28 = coord(7/25)
   

5. Reeve, L.H.; Han, H.; Brooks, A.D.: ¬The use of domain-specific concepts in biomedical text summarization (2007) 0.21

   0.21375035 = sum of:
     0.21375035 = product of:
       0.7633941 = sum of:
         0.108404 = weight(abstract_txt:summarization in 955) [ClassicSimilarity], result of:
           0.108404 = score(doc=955,freq=4.0), product of:
             0.121588126 = queryWeight, product of:
               1.0154442 = boost
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.016787691 = queryNorm
             0.89156735 = fieldWeight in 955, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               7.132539 = idf(docFreq=95, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.0467166 = weight(abstract_txt:evaluation in 955) [ClassicSimilarity], result of:
           0.0467166 = score(doc=955,freq=3.0), product of:
             0.0961977 = queryWeight, product of:
               1.2773439 = boost
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.016787691 = queryNorm
             0.48563117 = fieldWeight in 955, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               4.4860687 = idf(docFreq=1353, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.11100827 = weight(abstract_txt:rouge in 955) [ClassicSimilarity], result of:
           0.11100827 = score(doc=955,freq=1.0), product of:
             0.19608806 = queryWeight, product of:
               1.2895436 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.016787691 = queryNorm
             0.56611437 = fieldWeight in 955, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.0296602 = weight(abstract_txt:performance in 955) [ClassicSimilarity], result of:
           0.0296602 = score(doc=955,freq=1.0), product of:
             0.10248815 = queryWeight, product of:
               1.3184458 = boost
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.016787691 = queryNorm
             0.28940126 = fieldWeight in 955, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.63042 = idf(docFreq=1171, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.040140767 = weight(abstract_txt:semantic in 955) [ClassicSimilarity], result of:
           0.040140767 = score(doc=955,freq=1.0), product of:
             0.14354183 = queryWeight, product of:
               1.9109983 = boost
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.016787691 = queryNorm
             0.2796451 = fieldWeight in 955, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.4743214 = idf(docFreq=1369, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.13345383 = weight(abstract_txt:method in 955) [ClassicSimilarity], result of:
           0.13345383 = score(doc=955,freq=6.0), product of:
             0.19367395 = queryWeight, product of:
               2.563162 = boost
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.016787691 = queryNorm
             0.68906444 = fieldWeight in 955, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.50095 = idf(docFreq=1333, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
         0.2940104 = weight(abstract_txt:summaries in 955) [ClassicSimilarity], result of:
           0.2940104 = score(doc=955,freq=2.0), product of:
             0.4729328 = queryWeight, product of:
               4.0053473 = boost
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.016787691 = queryNorm
             0.62167484 = fieldWeight in 955, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.033448 = idf(docFreq=105, maxDocs=44218)
               0.0625 = fieldNorm(doc=955)
       0.28 = coord(7/25)