Document (#43012)

Author

Zou, J.
Thoma, G.
Antani, S.

Title

Unified deep neural network for segmentation and labeling of multipanel biomedical figures

Source

Journal of the Association for Information Science and Technology. 71(2020) no.11, S.1327-1340

Year

2020

Abstract

Recent efforts in biomedical visual question answering (VQA) research rely on combined information gathered from the image content and surrounding text supporting the figure. Biomedical journals are a rich source of information for such multimodal content indexing. For multipanel figures in these journals, it is critical to develop automatic figure panel splitting and label recognition algorithms to associate individual panels with text metadata in the figure caption and the body of the article. Challenges in this task include large variations in figure panel layout, label location, size, contrast to background, and so on. In this work, we propose a deep convolutional neural network, which splits the panels and recognizes the panel labels in a single step. Visual features are extracted from several layers at various depths of the backbone neural network and organized to form a feature pyramid. These features are fed into classification and regression networks to generate candidates of panels and their labels. These candidates are merged to create the final panel segmentation result through a beam search algorithm. We evaluated the proposed algorithm on the ImageCLEF data set and achieved better performance than the results reported in the literature. In order to thoroughly investigate the proposed algorithm, we also collected and annotated our own data set of 10,642 figures. The experiments, trained on 9,642 figures and evaluated on the remaining 1,000 figures, show that combining panel splitting and panel label recognition mutually benefit each other.

Content

https://asistdl.onlinelibrary.wiley.com/doi/10.1002/asi.24334.

Theme

Visualisierung

Field

Medizin

Similar documents (content)

1. Apostolova, E.; You, D.; Xue, Z.; Antani, S.; Demner-Fushman, D.; Thoma, G.R.: Image retrieval from scientific publications : text and image content processing to separate multipanel figures (2013) 0.40

   0.3986556 = sum of:
     0.3986556 = product of:
       1.42377 = sum of:
         0.08125382 = weight(abstract_txt:caption in 740) [ClassicSimilarity], result of:
           0.08125382 = score(doc=740,freq=2.0), product of:
             0.10503142 = queryWeight, product of:
               1.0274234 = boost
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.011679931 = queryNorm
             0.7736144 = fieldWeight in 740, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               8.752448 = idf(docFreq=18, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.0929788 = weight(abstract_txt:labels in 740) [ClassicSimilarity], result of:
           0.0929788 = score(doc=740,freq=2.0), product of:
             0.14477359 = queryWeight, product of:
               1.705884 = boost
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.011679931 = queryNorm
             0.64223593 = fieldWeight in 740, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.08564929 = weight(abstract_txt:segmentation in 740) [ClassicSimilarity], result of:
           0.08564929 = score(doc=740,freq=1.0), product of:
             0.17268683 = queryWeight, product of:
               1.8630925 = boost
               7.935687 = idf(docFreq=42, maxDocs=44218)
               0.011679931 = queryNorm
             0.49598044 = fieldWeight in 740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.935687 = idf(docFreq=42, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.18671186 = weight(abstract_txt:panels in 740) [ClassicSimilarity], result of:
           0.18671186 = score(doc=740,freq=1.0), product of:
             0.3323444 = queryWeight, product of:
               3.1655169 = boost
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.011679931 = queryNorm
             0.5618023 = fieldWeight in 740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.1656029 = weight(abstract_txt:figure in 740) [ClassicSimilarity], result of:
           0.1656029 = score(doc=740,freq=1.0), product of:
             0.3376748 = queryWeight, product of:
               3.6844199 = boost
               7.84674 = idf(docFreq=46, maxDocs=44218)
               0.011679931 = queryNorm
             0.49042124 = fieldWeight in 740, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.84674 = idf(docFreq=46, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.39142245 = weight(abstract_txt:figures in 740) [ClassicSimilarity], result of:
           0.39142245 = score(doc=740,freq=5.0), product of:
             0.37745383 = queryWeight, product of:
               4.355186 = boost
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.011679931 = queryNorm
             1.0370075 = fieldWeight in 740, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.4202213 = idf(docFreq=71, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
         0.42015082 = weight(abstract_txt:panel in 740) [ClassicSimilarity], result of:
           0.42015082 = score(doc=740,freq=3.0), product of:
             0.49855548 = queryWeight, product of:
               5.48305 = boost
               7.7848644 = idf(docFreq=49, maxDocs=44218)
               0.011679931 = queryNorm
             0.8427363 = fieldWeight in 740, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               7.7848644 = idf(docFreq=49, maxDocs=44218)
               0.0625 = fieldNorm(doc=740)
       0.28 = coord(7/25)
   

2. Ru, C.; Tang, J.; Li, S.; Xie, S.; Wang, T.: Using semantic similarity to reduce wrong labels in distant supervision for relation extraction (2018) 0.13

   0.12773585 = sum of:
     0.12773585 = product of:
       0.4561995 = sum of:
         0.022667792 = weight(abstract_txt:features in 5055) [ClassicSimilarity], result of:
           0.022667792 = score(doc=5055,freq=2.0), product of:
             0.05649884 = queryWeight, product of:
               1.0656745 = boost
               4.5391517 = idf(docFreq=1283, maxDocs=44218)
               0.011679931 = queryNorm
             0.4012081 = fieldWeight in 5055, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.5391517 = idf(docFreq=1283, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.06525728 = weight(abstract_txt:convolutional in 5055) [ClassicSimilarity], result of:
           0.06525728 = score(doc=5055,freq=1.0), product of:
             0.114336796 = queryWeight, product of:
               1.0719705 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.011679931 = queryNorm
             0.5707461 = fieldWeight in 5055, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.016783359 = weight(abstract_txt:proposed in 5055) [ClassicSimilarity], result of:
           0.016783359 = score(doc=5055,freq=1.0), product of:
             0.058258947 = queryWeight, product of:
               1.0821466 = boost
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.011679931 = queryNorm
             0.2880821 = fieldWeight in 5055, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.035935372 = weight(abstract_txt:network in 5055) [ClassicSimilarity], result of:
           0.035935372 = score(doc=5055,freq=2.0), product of:
             0.08793165 = queryWeight, product of:
               1.6282574 = boost
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.011679931 = queryNorm
             0.4086739 = fieldWeight in 5055, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.1470124 = weight(abstract_txt:labels in 5055) [ClassicSimilarity], result of:
           0.1470124 = score(doc=5055,freq=5.0), product of:
             0.14477359 = queryWeight, product of:
               1.705884 = boost
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.011679931 = queryNorm
             1.0154642 = fieldWeight in 5055, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               7.2660704 = idf(docFreq=83, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.04774467 = weight(abstract_txt:algorithm in 5055) [ClassicSimilarity], result of:
           0.04774467 = score(doc=5055,freq=1.0), product of:
             0.13389274 = queryWeight, product of:
               2.0092268 = boost
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.011679931 = queryNorm
             0.35658893 = fieldWeight in 5055, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.705423 = idf(docFreq=399, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
         0.12079865 = weight(abstract_txt:neural in 5055) [ClassicSimilarity], result of:
           0.12079865 = score(doc=5055,freq=2.0), product of:
             0.19732034 = queryWeight, product of:
               2.439138 = boost
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.011679931 = queryNorm
             0.6121956 = fieldWeight in 5055, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.0625 = fieldNorm(doc=5055)
       0.28 = coord(7/25)
   

3. Seeber, M.; Vlegels, J.; Cattaneo, M.: Conditions that do or do not disadvantage interdisciplinary research proposals in project evaluation (2022) 0.12

   0.11521425 = sum of:
     0.11521425 = product of:
       0.7200891 = sum of:
         0.008330465 = weight(abstract_txt:these in 636) [ClassicSimilarity], result of:
           0.008330465 = score(doc=636,freq=1.0), product of:
             0.041807413 = queryWeight, product of:
               1.1227345 = boost
               3.1881294 = idf(docFreq=4957, maxDocs=44218)
               0.011679931 = queryNorm
             0.19925809 = fieldWeight in 636, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               3.1881294 = idf(docFreq=4957, maxDocs=44218)
               0.0625 = fieldNorm(doc=636)
         0.04531249 = weight(abstract_txt:evaluated in 636) [ClassicSimilarity], result of:
           0.04531249 = score(doc=636,freq=2.0), product of:
             0.089655854 = queryWeight, product of:
               1.3424377 = boost
               5.7180014 = idf(docFreq=394, maxDocs=44218)
               0.011679931 = queryNorm
             0.5054047 = fieldWeight in 636, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               5.7180014 = idf(docFreq=394, maxDocs=44218)
               0.0625 = fieldNorm(doc=636)
         0.32339442 = weight(abstract_txt:panels in 636) [ClassicSimilarity], result of:
           0.32339442 = score(doc=636,freq=3.0), product of:
             0.3323444 = queryWeight, product of:
               3.1655169 = boost
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.011679931 = queryNorm
             0.97307014 = fieldWeight in 636, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.988837 = idf(docFreq=14, maxDocs=44218)
               0.0625 = fieldNorm(doc=636)
         0.34305167 = weight(abstract_txt:panel in 636) [ClassicSimilarity], result of:
           0.34305167 = score(doc=636,freq=2.0), product of:
             0.49855548 = queryWeight, product of:
               5.48305 = boost
               7.7848644 = idf(docFreq=49, maxDocs=44218)
               0.011679931 = queryNorm
             0.6880913 = fieldWeight in 636, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               7.7848644 = idf(docFreq=49, maxDocs=44218)
               0.0625 = fieldNorm(doc=636)
       0.16 = coord(4/25)
   

4. Huang, H.-H.; Wang, J.-J.; Chen, H.-H.: Implicit opinion analysis : extraction and polarity labelling (2017) 0.11

   0.11043692 = sum of:
     0.11043692 = product of:
       0.5521846 = sum of:
         0.09788592 = weight(abstract_txt:convolutional in 3820) [ClassicSimilarity], result of:
           0.09788592 = score(doc=3820,freq=1.0), product of:
             0.114336796 = queryWeight, product of:
               1.0719705 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.011679931 = queryNorm
             0.85611916 = fieldWeight in 3820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.09375 = fieldNorm(doc=3820)
         0.07340217 = weight(abstract_txt:deep in 3820) [ClassicSimilarity], result of:
           0.07340217 = score(doc=3820,freq=1.0), product of:
             0.11890181 = queryWeight, product of:
               1.5459628 = boost
               6.5848994 = idf(docFreq=165, maxDocs=44218)
               0.011679931 = queryNorm
             0.6173343 = fieldWeight in 3820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               6.5848994 = idf(docFreq=165, maxDocs=44218)
               0.09375 = fieldNorm(doc=3820)
         0.05390306 = weight(abstract_txt:network in 3820) [ClassicSimilarity], result of:
           0.05390306 = score(doc=3820,freq=2.0), product of:
             0.08793165 = queryWeight, product of:
               1.6282574 = boost
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.011679931 = queryNorm
             0.6130109 = fieldWeight in 3820, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.09375 = fieldNorm(doc=3820)
         0.18119797 = weight(abstract_txt:neural in 3820) [ClassicSimilarity], result of:
           0.18119797 = score(doc=3820,freq=2.0), product of:
             0.19732034 = queryWeight, product of:
               2.439138 = boost
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.011679931 = queryNorm
             0.9182934 = fieldWeight in 3820, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.09375 = fieldNorm(doc=3820)
         0.14579543 = weight(abstract_txt:label in 3820) [ClassicSimilarity], result of:
           0.14579543 = score(doc=3820,freq=1.0), product of:
             0.2150679 = queryWeight, product of:
               2.546468 = boost
               7.230979 = idf(docFreq=86, maxDocs=44218)
               0.011679931 = queryNorm
             0.67790425 = fieldWeight in 3820, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               7.230979 = idf(docFreq=86, maxDocs=44218)
               0.09375 = fieldNorm(doc=3820)
       0.2 = coord(5/25)
   

5. Wang, P.; Li, X.: Assessing the quality of information on Wikipedia : a deep-learning approach (2020) 0.10

   0.10440247 = sum of:
     0.10440247 = product of:
       0.37286595 = sum of:
         0.022667792 = weight(abstract_txt:features in 5505) [ClassicSimilarity], result of:
           0.022667792 = score(doc=5505,freq=2.0), product of:
             0.05649884 = queryWeight, product of:
               1.0656745 = boost
               4.5391517 = idf(docFreq=1283, maxDocs=44218)
               0.011679931 = queryNorm
             0.4012081 = fieldWeight in 5505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.5391517 = idf(docFreq=1283, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.06525728 = weight(abstract_txt:convolutional in 5505) [ClassicSimilarity], result of:
           0.06525728 = score(doc=5505,freq=1.0), product of:
             0.114336796 = queryWeight, product of:
               1.0719705 = boost
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.011679931 = queryNorm
             0.5707461 = fieldWeight in 5505, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.131938 = idf(docFreq=12, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.016783359 = weight(abstract_txt:proposed in 5505) [ClassicSimilarity], result of:
           0.016783359 = score(doc=5505,freq=1.0), product of:
             0.058258947 = queryWeight, product of:
               1.0821466 = boost
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.011679931 = queryNorm
             0.2880821 = fieldWeight in 5505, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.6093135 = idf(docFreq=1196, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.011781056 = weight(abstract_txt:these in 5505) [ClassicSimilarity], result of:
           0.011781056 = score(doc=5505,freq=2.0), product of:
             0.041807413 = queryWeight, product of:
               1.1227345 = boost
               3.1881294 = idf(docFreq=4957, maxDocs=44218)
               0.011679931 = queryNorm
             0.28179348 = fieldWeight in 5505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               3.1881294 = idf(docFreq=4957, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.08475752 = weight(abstract_txt:deep in 5505) [ClassicSimilarity], result of:
           0.08475752 = score(doc=5505,freq=3.0), product of:
             0.11890181 = queryWeight, product of:
               1.5459628 = boost
               6.5848994 = idf(docFreq=165, maxDocs=44218)
               0.011679931 = queryNorm
             0.71283627 = fieldWeight in 5505, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               6.5848994 = idf(docFreq=165, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.05082029 = weight(abstract_txt:network in 5505) [ClassicSimilarity], result of:
           0.05082029 = score(doc=5505,freq=4.0), product of:
             0.08793165 = queryWeight, product of:
               1.6282574 = boost
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.011679931 = queryNorm
             0.5779522 = fieldWeight in 5505, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.6236176 = idf(docFreq=1179, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
         0.12079865 = weight(abstract_txt:neural in 5505) [ClassicSimilarity], result of:
           0.12079865 = score(doc=5505,freq=2.0), product of:
             0.19732034 = queryWeight, product of:
               2.439138 = boost
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.011679931 = queryNorm
             0.6121956 = fieldWeight in 5505, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               6.926203 = idf(docFreq=117, maxDocs=44218)
               0.0625 = fieldNorm(doc=5505)
       0.28 = coord(7/25)