More Like this

1. MedICaT: A Dataset of Medical Images, Captions, and Textual References

   https://doi.org/10.18653/v1/2020.findings-emnlp.191  

   Subramanian, Sanjay; Wang, Lucy Lu; Bogin, Ben; Mehta, Sachin; van Zuylen, Madeleine; Parasa, Sravanthi; Singh, Sameer; Gardner, Matt; Hajishirzi, Hannaneh (January 2020, Findings of the Association for Computational Linguistics: EMNLP)

   Understanding the relationship between figures and text is key to scientific document understanding. Medical figures in particular are quite complex, often consisting of several subfigures (75% of figures in our dataset), with detailed text describing their content. Previous work studying figures in scientific papers focused on classifying figure content rather than understanding how images relate to the text. To address challenges in figure retrieval and figure-to-text alignment, we introduce MedICaT, a dataset of medical images in context. MedICaT consists of 217K images from 131K open access biomedical papers, and includes captions, inline references for 74% of figures, and manually annotated subfigures and subcaptions for a subset of figures. Using MedICaT, we introduce the task of subfigure to subcaption alignment in compound figures and demonstrate the utility of inline references in image-text matching. Our data and code can be accessed at https://github.com/allenai/medicat. 

   more » « less
2. A Guided-Inquiry Activity for Introducing Students to Figures from Primary Scientific Literature

   https://doi.org/10.1021/acs.jchemed.2c00605  

   Maxwell, Danielle N.; Spencer, Jeffrey L.; Teich, Ethan A.; Cooke, Madeline; Fromwiller, Braeden; Peterson, Nathan; Nicholas-Figueroa, Linda; Shultz, Ginger V.; Pratt, Kerri A. (May 2023, Journal of Chemical Education)

   Holme, Thomas (Ed.)

   Reading and understanding scientific literature is an essential skill for any scientist to learn. While students’ scientific literacy can be improved by reading research articles, an article’s technical language and structure can hinder students’ understanding of the scientific material. Furthermore, many students struggle with interpreting graphs and other models of data commonly found in scientific literature. To introduce students to scientific literature and promote improved understanding of data and graphs, we developed a guided-inquiry activity adapted from a research article on snow chemistry and implemented it in a general chemistry laboratory course. Here, we describe how we adapted figures from the primary literature source and developed questions to scaffold the guided-inquiry activity. Results from semi-structured qualitative interviews suggest that students learn about snow chemistry processes and engage in scientific practices, including data analysis and interpretation, through this activity. This activity is applicable in other introductory science courses as educators can adapt most scientific articles into a guided-inquiry activity. 

   more » « less
3. Identifying the Central Figure of a Scientific Paper

   https://doi.org/10.1109/ICDAR.2019.00173  

   Yang, Sean T.; Lee, Po-Shen; Kazakova, Lia; Joshi, Abhishek; Oh, Bum Mook; West, Jevin D.; Howe, Bill (September 2019, 2019 International Conference on Document Analysis and Recognition (ICDAR))

   Publishers are increasingly using graphical abstracts to facilitate scientific search, especially across disciplinary boundaries. They are presented on various media, easily shared and information rich. However, very small amount of scientific publications are equipped with graphical abstracts. What can we do with the vast majority of papers with no selected graphical abstract? In this paper, we first hypothesize that scientific papers actually include a "central figure" that serve as a graphical abstract. These figures convey the key results and provide a visual identity for the paper. Using survey data collected from 6,263 authors regarding 8,353 papers over 15 years, we find that over 87% of papers are considered to contain a central figure, and that these central figures are primarily used to summarize important results, explain the key methods, or provide additional discussion. We then train a model to automatically recognize the central figure, achieving top-3 accuracy of 78% and exact match accuracy of 34%. We find that the primary boost in accuracy comes from figure captions that resemble the abstract. We make all our data and results publicly available at https://github.com/viziometrics/centraul_figure. Our goal is to automate central figure identification to improve search engine performance and to help scientists connect ideas across the literature. 

   more » « less
4. Raman Match: Application for automated identification of minerals from Raman spectroscopy data

   https://doi.org/10.2138/am-2023-9227  

   Berrada, Meryem; McFall, Alan; Chen, Bin (January 2025, American Mineralogist)

   Kung, Jennifer (Ed.)

   Abstract Raman spectroscopy is a rapid, nondestructive analysis technique used in various scientific disciplines, including mineralogy, chemistry, materials science, and biology. The analysis of Raman spectra and the identification of specific substances in unknown samples can be complex and time-consuming due to the large database of Raman spectra. The Raman Match application was developed to simplify and automate the sample identification process through a search and match method. The application integrates the well-established RRUFF Raman database with the Python programming language. It provides a user-friendly graphical interface to load Raman spectra, identify and fit peaks, match peaks to the reference libraries, visualize the results, and generate publication-ready figures. The application offers a swift and automated method for mineral identification using Raman spectroscopy in laboratory and field settings and during planetary exploration missions to extraterrestrial environments with constraints on time and resources. 

   more » « less
5. Scientific figures interpreted by ChatGPT: strengths in plot recognition and limits in color perception

   https://doi.org/10.1038/s41698-024-00576-z  

   Wang, Jinge; Ye, Qing; Liu, Li; Guo, Nancy Lan; Hu, Gangqing (December 2024, npj Precision Oncology)

   Abstract Emerging studies underscore the promising capabilities of large language model-based chatbots in conducting basic bioinformatics data analyses. The recent feature of accepting image inputs by ChatGPT, also known as GPT-4V(ision), motivated us to explore its efficacy in deciphering bioinformatics scientific figures. Our evaluation with examples in cancer research, including sequencing data analysis, multimodal network-based drug repositioning, and tumor clonal evolution, revealed that ChatGPT can proficiently explain different plot types and apply biological knowledge to enrich interpretations. However, it struggled to provide accurate interpretations when color perception and quantitative analysis of visual elements were involved. Furthermore, while the chatbot can draft figure legends and summarize findings from the figures, stringent proofreading is imperative to ensure the accuracy and reliability of the content. 

   more » « less