More Like this

1. ICPR 2020 - Competition on Harvesting Raw Tables from Infographics

   https://doi.org/10.1007/978-3-030-68793-9_27  

   Davila, Kenny; Tensmeyer, Chris; Shekhar, Sumit; Singh, Hrituraj; Setlur, Srirangaraj; Govindaraju, Venu. (February 2021, Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021)

   Del Bimbo, Alberto; Cucchiara, Rita; Sclaroff, Stan; Farinella, Giovanni M; Mei, Tao; Bertini, Marc; Escalante, Hugo J; Vezzani, Roberto. (Ed.)

   This work summarizes the results of the second Competition on Harvesting Raw Tables from Infographics (ICPR 2020 CHART-Infographics). Chart Recognition is difficult and multifaceted, so for this competition we divide the process into the following tasks: Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We provided two sets of datasets for training and evaluation of the participant submissions. The first set is based on synthetic charts (Adobe Synth) generated from real data sources using matplotlib. The second one is based on manually annotated charts extracted from the Open Access section of the PubMed Central (UB PMC). More than 25 teams registered out of which 7 submitted results for different tasks of the competition. While results on synthetic data are near perfect at times, the same models still have room to improve when it comes to data extraction from real charts. The data, annotation tools, and evaluation scripts have been publicly released for academic use. 

   more » « less
2. Parsing and Summarizing Infographics with Synthetically Trained Icon Detection

   https://doi.org/10.1109/PacificVis52677.2021.00012  

   Madan, Spandan; Bylinskii, Zoya; Nobre, Carolina; Tancik, Matthew; Recasens, Adria; Zhong, Kimberli; Alsheikh, Sami; Oliva, Aude; Durand, Fredo; Pfister, Hanspeter (April 2021, 2021 IEEE 14th Pacific Visualization Symposium (PacificVis))

   null (Ed.)

   Widely used in news, business, and educational media, infographics are handcrafted to effectively communicate messages about complex and often abstract topics including `ways to conserve the environment' and `coronavirus prevention'. The computational understanding of infographics required for future applications like automatic captioning, summarization, search, and question-answering, will depend on being able to parse the visual and textual elements contained within. However, being composed of stylistically and semantically diverse visual and textual elements, infographics pose challenges for current A.I. systems. While automatic text extraction works reasonably well on infographics, standard object detection algorithms fail to identify the stand-alone visual elements in infographics that we refer to as `icons'. In this paper, we propose a novel approach to train an object detector using synthetically-generated data, and show that it succeeds at generalizing to detecting icons within in-the-wild infographics. We further pair our icon detection approach with an icon classifier and a state-of-the-art text detector to demonstrate three demo applications: topic prediction, multi-modal summarization, and multi-modal search. Parsing the visual and textual elements within infographics provides us with the first steps towards automatic infographic understanding. 

   more » « less
3. Chart Mining: A Survey of Methods for Automated Chart Analysis

   https://doi.org/10.1109/TPAMI.2020.2992028  

   Davila, Kenny; Setlur, Srirangaraj; Doermann, David; Bhargava, Urala Kota; Govindaraju, Venu (May 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence)

   null (Ed.)

   Charts are useful communication tools for the presentation of data in a visually appealing format that facilitates comprehension. There have been many studies dedicated to chart mining, which refers to the process of automatic detection, extraction and analysis of charts to reproduce the tabular data that was originally used to create them. By allowing access to data which might not be available in other formats, chart mining facilitates the creation of many downstream applications. This paper presents a comprehensive survey of approaches across all components of the automated chart mining pipeline such as (i) automated extraction of charts from documents; (ii) processing of multi-panel charts; (iii) automatic image classifiers to collect chart images at scale; (iv) automated extraction of data from each chart image, for popular chart types as well as selected specialized classes; (v) applications of chart mining; and (vi) datasets for training and evaluation, and the methods that were used to build them. Finally, we summarize the main trends found in the literature and provide pointers to areas for further research in chart mining. 

   more » « less
4. DPRL Systems in the CLEF 2020 ARQMath Lab

   Mansouri, Behrooz; Oard, Douglas W.; Zanibbi, Richard (January 2020, Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum)

   This paper describes the participation of the Document and Pattern Recognition Lab from the Rochester Institute of Technology in the CLEF 2020 ARQMath lab. There are two tasks defined for ARQMath: (1) Question Answering, and (2) Formula Retrieval. Four runs were submitted for Task 1 using systems that take advantage of text and formula embeddings. For Task 2, three runs were submitted: one uses only formula embedding, another uses formula and text embeddings, and the final one uses formula embedding followed by re-ranking results by tree-edit distance. The Task 2 runs yielded strong results, the Task 1 results were less competitive. 

   more » « less
5. Knowing what to look for: A Fact-Evidence Reasoning Framework for Decoding Communicative Visualization

   https://doi.org/10.1109/VIS47514.2020.00053  

   Vaidya, Sahaj; Dasgupta, Aritra (October 2020, 2020 IEEE Visualization Conference (VIS))

   null (Ed.)

   Despite the widespread use of communicative charts as a medium for scientific communication, we lack a systematic understanding of how well the charts fulfill the goals of effective visual communication. Existing research mostly focuses on the means, i.e. the encoding principles, and not the end, i.e. the key takeaway of a chart. To address this gap, we start from the first principles and aim to answer the fundamental question: how can we describe the message of a scientific chart? We contribute a fact-evidence reasoning framework (FaEvR) by augmenting the conventional visualization pipeline with the stages of gathering and associating evidence for decoding the facts presented in a chart. We apply the resulting classification scheme of fact and evidence on a collection of 500 charts collected from publications in multiple science domains. We demonstrate the practical applications of FaEvR in calibrating task complexity and detecting barriers towards chart interpretability. 

   more » « less