More Like this

1. ICDAR 2019 Competition on Harvesting Raw Tables from Infographics (CHART-Infographics)

   https://doi.org/10.1109/ICDAR.2019.00203  

   Davila, Kenny; Kota, Bhargava Urala; Setlur, Srirangaraj; Govindaraju, Venu; Tensmeyer, Christopher; Shekhar, Sumit; Chaudhry, Ritwick (September 2019, 2019 International Conference on Document Analysis and Recognition (ICDAR))

   This work summarizes the results of the first Competition on Harvesting Raw Tables from Infographics (ICDAR 2019 CHART-Infographics). The complex process of automatic chart recognition is divided into multiple tasks for the purpose of this competition, including Chart Image Classification (Task 1), Text Detection and Recognition (Task 2), Text Role Classification (Task 3), Axis Analysis (Task 4), Legend Analysis (Task 5), Plot Element Detection and Classification (Task 6.a), Data Extraction (Task 6.b), and End-to-End Data Extraction (Task 7). We provided a large synthetic training set and evaluated submitted systems using newly proposed metrics on both synthetic charts and manually-annotated real charts taken from scientific literature. A total of 8 groups registered for the competition out of which 5 submitted results for tasks 1-5. The results show that some tasks can be performed highly accurately on synthetic data, but all systems did not perform as well on real world charts. The data, annotation tools, and evaluation scripts have been publicly released for academic use. 

   more » « less
2. Chart Mining: A Survey of Methods for Automated Chart Analysis

   https://doi.org/10.1109/TPAMI.2020.2992028  

   Davila, Kenny; Setlur, Srirangaraj; Doermann, David; Bhargava, Urala Kota; Govindaraju, Venu (May 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence)

   null (Ed.)

   Charts are useful communication tools for the presentation of data in a visually appealing format that facilitates comprehension. There have been many studies dedicated to chart mining, which refers to the process of automatic detection, extraction and analysis of charts to reproduce the tabular data that was originally used to create them. By allowing access to data which might not be available in other formats, chart mining facilitates the creation of many downstream applications. This paper presents a comprehensive survey of approaches across all components of the automated chart mining pipeline such as (i) automated extraction of charts from documents; (ii) processing of multi-panel charts; (iii) automatic image classifiers to collect chart images at scale; (iv) automated extraction of data from each chart image, for popular chart types as well as selected specialized classes; (v) applications of chart mining; and (vi) datasets for training and evaluation, and the methods that were used to build them. Finally, we summarize the main trends found in the literature and provide pointers to areas for further research in chart mining. 

   more » « less
3. MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems

   Zhu, Zifeng; Jia, Mengzhao; Zhang, Zhihan; Li, Lang; Jiang, Meng (April 2025, Association for Computational Linguistics)

   Multimodal Large Language Models (MLLMs) have demonstrated impressive abilities across various tasks, including visual question answering and chart comprehension, yet existing benchmarks for chart-related tasks fall short in capturing the complexity of real-world multi-chart scenarios. Current benchmarks primarily focus on single-chart tasks, neglecting the multi-hop reasoning required to extract and integrate information from multiple charts, which is essential in practical applications. To fill this gap, we introduce MultiChartQA, a benchmark that evaluates MLLMs’ capabilities in four key areas: direct question answering, parallel question answering, comparative reasoning, and sequential reasoning. Our evaluation of a wide range of MLLMs reveals significant performance gaps compared to humans. These results highlight the challenges in multi-chart comprehension and the potential of MultiChartQA to drive advancements in this field. Our code and data are available at https://github.com/Zivenzhu/Multi-chart-QA. 

   more » « less
4. Knowing what to look for: A Fact-Evidence Reasoning Framework for Decoding Communicative Visualization

   https://doi.org/10.1109/VIS47514.2020.00053  

   Vaidya, Sahaj; Dasgupta, Aritra (October 2020, 2020 IEEE Visualization Conference (VIS))

   null (Ed.)

   Despite the widespread use of communicative charts as a medium for scientific communication, we lack a systematic understanding of how well the charts fulfill the goals of effective visual communication. Existing research mostly focuses on the means, i.e. the encoding principles, and not the end, i.e. the key takeaway of a chart. To address this gap, we start from the first principles and aim to answer the fundamental question: how can we describe the message of a scientific chart? We contribute a fact-evidence reasoning framework (FaEvR) by augmenting the conventional visualization pipeline with the stages of gathering and associating evidence for decoding the facts presented in a chart. We apply the resulting classification scheme of fact and evidence on a collection of 500 charts collected from publications in multiple science domains. We demonstrate the practical applications of FaEvR in calibrating task complexity and detecting barriers towards chart interpretability. 

   more » « less
5. A Multi-Level Optimization Framework for End-to-End Text Augmentation

   https://doi.org/10.1162/tacl_a_00464  

   Somayajula, Sai Ashish; Song, Linfeng; Xie, Pengtao (January 2022, Transactions of the Association for Computational Linguistics)

   Abstract Text augmentation is an effective technique in alleviating overfitting in NLP tasks. In existing methods, text augmentation and downstream tasks are mostly performed separately. As a result, the augmented texts may not be optimal to train the downstream model. To address this problem, we propose a three-level optimization framework to perform text augmentation and the downstream task end-to- end. The augmentation model is trained in a way tailored to the downstream task. Our framework consists of three learning stages. A text summarization model is trained to perform data augmentation at the first stage. Each summarization example is associated with a weight to account for its domain difference with the text classification data. At the second stage, we use the model trained at the first stage to perform text augmentation and train a text classification model on the augmented texts. At the third stage, we evaluate the text classification model trained at the second stage and update weights of summarization examples by minimizing the validation loss. These three stages are performed end-to-end. We evaluate our method on several text classification datasets where the results demonstrate the effectiveness of our method. Code is available at https://github.com/Sai-Ashish/End-to-End-Text-Augmentation. 

   more » « less