skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: The Risks of Ranking: Revisiting Graphical Perception to Model Individual Differences in Visualization Performance
Graphical perception studies typically measure visualization encoding effectiveness using the error of an “average observer”, leading to canonical rankings of encodings for numerical attributes: e.g., position > area > angle > volume. Yet different people may vary in their ability to read different visualization types, leading to variance in this ranking across individuals not captured by population-level metrics using “average observer” models. One way we can bridge this gap is by recasting classic visual perception tasks as tools for assessing individual performance, in addition to overall visualization performance. In this article we replicate and extend Cleveland and McGill's graphical comparison experiment using Bayesian multilevel regression, using these models to explore individual differences in visualization skill from multiple perspectives. The results from experiments and modeling indicate that some people show patterns of accuracy that credibly deviate from the canonical rankings of visualization effectiveness. We discuss implications of these findings, such as a need for new ways to communicate visualization effectiveness to designers, how patterns in individuals’ responses may show systematic biases and strategies in visualization judgment, and how recasting classic visual perception tasks as tools for assessing individual performance may offer new ways to quantify aspects of visualization literacy. Experiment data, source code, and analysis scripts are available at the following repository: https://osf.io/8ub7t/?view_only=9be4798797404a4397be3c6fc2a68cc0 .  more » « less
Award ID(s):
1815790 2120750
PAR ID:
10504925
Author(s) / Creator(s):
; ; ; ; ; ; ;
Publisher / Repository:
IEEE
Date Published:
Journal Name:
IEEE Transactions on Visualization and Computer Graphics
Volume:
30
Issue:
3
ISSN:
1077-2626
Page Range / eLocation ID:
1756 to 1771
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Jodie Jenkinson, Susan Keen (Ed.)
    While visual literacy has been identified as a foundational skill in life science education, there are many challenges in teaching and assessing biomolecular visualization skills. Among these are the lack of consensus about what constitutes competence and limited understanding of student and instructor perceptions of visual literacy tasks. In this study, we administered a set of biomolecular visualization assessments, developed as part of the BioMolViz project, to both students and instructors at multiple institutions and compared their perceptions of task difficulty. We then analyzed our findings using a mixed-methods approach. Quantitative analysis was used to answer the following research questions: (1) Which assessment items exhibit statistically significant disparities or agreements in perceptions of difficulty between instructors and students? (2) Do these perceptions persist when controlling for race/ethnicity and gender? and (3) How does student perception of difficulty relate to performance? Qualitative analysis of open-ended comments was used to identify predominant themes related to visual problem solving. The results show that perceptions of difficulty significantly differ between students and instructors and that students’ performance is a significant predictor of their perception of difficulty. Overall, this study underscores the need to incorporate deliberate instruction in visualization into undergraduate life science curricula to improve student ability in this area. Accordingly, we offer recommendations to promote visual literacy skills in the classroom. 
    more » « less
  2. Abstract Decisions involving algorithmic rankings affect our lives in many ways, from product recommendations, receiving scholarships, to securing jobs. While tools have been developed for interactively constructing fair consensus rankings from a handful of rankings, addressing the more complex real‐world scenario— where diverse opinions are represented by a larger collection of rankings— remains a challenge. In this paper, we address these challenges by reformulating the exploration of rankings as a dimension reduction problem in a system called FairSpace. FairSpace provides new views, including Fair Divergence View and Cluster Views, by juxtaposing fairness metrics of different local and alternative global consensus rankings to aid ranking analysis tasks. We illustrate the effectiveness of FairSpace through a series of use cases, demonstrating via interactive workflows that users are empowered to create local consensuses by grouping rankings similar in their fairness or utility properties, followed by hierarchically aggregating local consensuses into a global consensus through direct manipulation. We discuss how FairSpace opens the possibility for advances in dimension reduction visualization to benefit the research area of supporting fair decision‐making in ranking based decision‐making contexts. Code, datasets and demo video available at:osf.io/d7cwk 
    more » « less
  3. Imagine pouring a box of granola into a bowl. Are you considering hundreds of individual chunks or the motion of the group as a whole? Human perceptual limits suggest we cannot be representing the individuals, implying we simulate ensembles of objects. If true, we would need to represent group physical properties beyond individual aggregates, similar to perceiving ensemble properties like color, size, or facial expression. Here we investigate whether people do hold ensemble representations of mass, using tasks in which participants watch a video of a single marble or set of marbles falling onto an elastic cloth and judge the individual or average mass. We find first that people better judge average masses than individual masses, then find evidence that the better ensemble judgments are not just due to aggregating information from individual marbles. Together, this supports the concept of ensemble perception in intuitive physics, extending our understanding of how people represent and simulate sets of objects. 
    more » « less
  4. There exists a gap between visualization design guidelines and their application in visualization tools. While empirical studies can provide design guidance, we lack a formal framework for representing design knowledge, integrating results across studies, and applying this knowledge in automated design tools that promote effective encodings and facilitate visual exploration. We propose modeling visualization design knowledge as a collection of constraints, in conjunction with a method to learn weights for soft constraints from experimental data. Using constraints, we can take theoretical design knowledge and express it in a concrete, extensible, and testable form: the resulting models can recommend visualization designs and can easily be augmented with additional constraints or updated weights. We implement our approach in Draco, a constraint-based system based on Answer Set Programming (ASP). We demonstrate how to construct increasingly sophisticated automated visualization design systems, including systems based on weights learned directly from the results of graphical perception experiments. 
    more » « less
  5. There exists a gap between visualization design guidelines and their application in visualization tools. While empirical studies can provide design guidance, we lack a formal framework for representing design knowledge, integrating results across studies, and applying this knowledge in automated design tools that promote effective encodings and facilitate visual exploration. We propose modeling visualization design knowledge as a collection of constraints, in conjunction with a method to learn weights for soft constraints from experimental data. Using constraints, we can take theoretical design knowledge and express it in a concrete, extensible, and testable form: the resulting models can recommend visualization designs and can easily be augmented with additional constraints or updated weights. We implement our approach in Draco, a constraint-based system based on Answer Set Programming (ASP). We demonstrate how to construct increasingly sophisticated automated visualization design systems, including systems based on weights learned directly from the results of graphical perception experiments. 
    more » « less