More Like this

1. Evaluating convergence between two data visualization literacy assessments

   https://doi.org/10.1186/s41235-025-00622-9  

   Brockbank, Erik; Verma, Arnav; Lloyd, Hannah; Huey, Holly; Padilla, Lace; Fan, Judith_E (April 2025, Cognitive Research: Principles and Implications)

   Abstract Data visualizations play a crucial role in communicating patterns in quantitative data, making data visualization literacy a key target of STEM education. However, it is currently unclear to what degree different assessments of data visualization literacy measure the same underlying constructs. Here, we administered two widely used graph comprehension assessments (Galesic and Garcia-Retamero in Med Dec Mak 31:444–457, 2011; Lee et al. in IEEE Trans Vis Comput Graph 235:51–560, 2016) to both a university-based convenience sample and a demographically representative sample of adult participants in the USA (N=1,113). Our analysis of individual variability in test performance suggests that overall scores are correlated between assessments and associated with the amount of prior coursework in mathematics. However, further exploration of individual error patterns suggests that these assessments probe somewhat distinct components of data visualization literacy, and we do not find evidence that these components correspond to the categories that guided the design of either test (e.g., questions that require retrieving values rather than making comparisons). Together, these findings suggest opportunities for development of more comprehensive assessments of data visualization literacy that are organized by components that better account for detailed behavioral patterns. 

   more » « less
2. Seeing Eye to Eye? Comparing Faculty and Student Perceptions of Biomolecular Visualization Assessments

   https://doi.org/10.3390/educsci14010094  

   Beckham, Josh T.; Dries, Daniel R.; Hall, Bonnie L.; Mitton-Fry, Rachel M.; Engelman, Shelly; Burch, Charmita; Acevedo, Roderico; Mertz, Pamela S.; Vardar-Ulu, Didem; Agrawal, Swati; et al (January 2024, Education Sciences)

   Jodie Jenkinson, Susan Keen (Ed.)

   While visual literacy has been identified as a foundational skill in life science education, there are many challenges in teaching and assessing biomolecular visualization skills. Among these are the lack of consensus about what constitutes competence and limited understanding of student and instructor perceptions of visual literacy tasks. In this study, we administered a set of biomolecular visualization assessments, developed as part of the BioMolViz project, to both students and instructors at multiple institutions and compared their perceptions of task difficulty. We then analyzed our findings using a mixed-methods approach. Quantitative analysis was used to answer the following research questions: (1) Which assessment items exhibit statistically significant disparities or agreements in perceptions of difficulty between instructors and students? (2) Do these perceptions persist when controlling for race/ethnicity and gender? and (3) How does student perception of difficulty relate to performance? Qualitative analysis of open-ended comments was used to identify predominant themes related to visual problem solving. The results show that perceptions of difficulty significantly differ between students and instructors and that students’ performance is a significant predictor of their perception of difficulty. Overall, this study underscores the need to incorporate deliberate instruction in visualization into undergraduate life science curricula to improve student ability in this area. Accordingly, we offer recommendations to promote visual literacy skills in the classroom. 

   more » « less
3. Benchmarking Visual Language Models on Standardized Visualization Literacy Tests

   https://doi.org/10.1111/cgf.70137  

   Pandey, Saugat; Ottley, Alvitta (May 2025, Computer Graphics Forum)

   Abstract The increasing integration of Visual Language Models (VLMs) into visualization systems demands a comprehensive understanding of their visual interpretation capabilities and constraints. While existing research has examined individual models, systematic comparisons of VLMs' visualization literacy remain unexplored. We bridge this gap through a rigorous, first‐of‐its‐kind evaluation of four leading VLMs (GPT‐4, Claude, Gemini, and Llama) using standardized assessments: the Visualization Literacy Assessment Test (VLAT) and Critical Thinking Assessment for Literacy in Visualizations (CALVI). Our methodology uniquely combines randomized trials with structured prompting techniques to control for order effects and response variability ‐ a critical consideration overlooked in many VLM evaluations. Our analysis reveals that while specific models demonstrate competence in basic chart interpretation (Claude achieving 67.9% accuracy on VLAT), all models exhibit substantial difficulties in identifying misleading visualization elements (maximum 30.0% accuracy on CALVI). We uncover distinct performance patterns: strong capabilities in interpreting conventional charts like line charts (76‐96% accuracy) and detecting hierarchical structures (80‐100% accuracy), but consistent difficulties with data‐dense visualizations involving multiple encodings (bubble charts: 18.6‐61.4%) and anomaly detection (25‐30% accuracy). Significantly, we observe distinct uncertainty management behavior across models, with Gemini displaying heightened caution (22.5% question omission) compared to others (7‐8%). These findings provide crucial insights for the visualization community by establishing reliable VLM evaluation benchmarks, identifying areas where current models fall short, and highlighting the need for targeted improvements in VLM architectures for visualization tasks. To promote reproducibility, encourage further research, and facilitate benchmarking of future VLMs, our complete evaluation framework, including code, prompts, and analysis scripts, is available athttps://github.com/washuvis/VisLit‐VLM‐Eval. 

   more » « less
4. Abstract 1286 Voicing Visual Literacy: Student Interviews to Improve Biomolecular Visualization Assessments

   https://doi.org/10.1016/j.jbc.2025.108576  

   Burch, Charmita; Beckham, Josh; Procko, Kristen; Vardar-Ulu, Didem; Genova, Lauren; Fox, Kristin; Mitton-Fry, Rachel; Acevedo, Roderico; Steere, Pamela (May 2025, Journal of Biological Chemistry)

   For a decade, BioMolViz has been developing tools to improve visual literacy instruction. In collaboration with the biochemistry and molecular biology (BMB) education community, our group authored a Biomolecular Visualization Framework to assess visual literacy skills and used the framework’s learning objectives in the backward design of assessments. Our validation process, which includes iterative revision by our working group of faculty, expert panel review, and large-scale classroom testing, has produced a subset of validated assessments which are available in our online repository, the BioMolViz Library. Nearly 200 assessments are now moving through the earlier phases of our validation process. With an eye always on inclusivity, we used our large-scale field testing data to examine performance trends. Upon observing some differences in performance that correlated with gender and race, we organized semi-structured interviews with small groups of undergraduate students to further evaluate our assessments. Disaggregating students into groups by gender, we asked students to share initial impressions and engage in collaborative reflection on their problem solving strategies. As we thematically code our interview transcripts, which include male and female groups from three U.S.-based institutions, we seek to further improve the clarity of our assessments, while exploring approaches to problem solving that may uncover demographic-related differences and make visual literacy more inclusive for all learners. 

   more » « less
5. Stealth Literacy Assessments via Educational Games

   https://doi.org/10.3390/computers12070130  

   Fang, Ying; Li, Tong; Huynh, Linh; Christhilf, Katerina; Roscoe, Rod D.; McNamara, Danielle S. (July 2023, Computers)

   Literacy assessment is essential for effective literacy instruction and training. However, traditional paper-based literacy assessments are typically decontextualized and may cause stress and anxiety for test takers. In contrast, serious games and game environments allow for the assessment of literacy in more authentic and engaging ways, which has some potential to increase the assessment’s validity and reliability. The primary objective of this study is to examine the feasibility of a novel approach for stealthily assessing literacy skills using games in an intelligent tutoring system (ITS) designed for reading comprehension strategy training. We investigated the degree to which learners’ game performance and enjoyment predicted their scores on standardized reading tests. Amazon Mechanical Turk participants (n = 211) played three games in iSTART and self-reported their level of game enjoyment after each game. Participants also completed the Gates–MacGinitie Reading Test (GMRT), which includes vocabulary knowledge and reading comprehension measures. The results indicated that participants’ performance in each game as well as the combined performance across all three games predicted their literacy skills. However, the relations between game enjoyment and literacy skills varied across games. These findings suggest the potential of leveraging serious games to assess students’ literacy skills and improve the adaptivity of game-based learning environments. 

   more » « less