An official website of the United States government
Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test ( vlat ), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests ( cats ) for visualization literacy, a-vlat and a-calvi , which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of a-calvi via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATS via four online studies. We discuss practical recommendations for using our CATS and opportunities for further customization to leverage the full potential of adaptive assessments. All supplemental materials are available at https://osf.io/a6258/ .
more »
« less
The numeric understanding measures: Developing and validating adaptive and nonadaptive numeracy scales
Abstract Numeracy—the ability to understand and use numeric information—is linked to good decision-making. Several problems exist with current numeracy measures, however. Depending on the participant sample, some existing measures are too easy or too hard; also, established measures often contain items well-known to participants. The current article aimed to develop new numeric understanding measures (NUMs) including a 1-item (1-NUM), 4-item (4-NUM), and 4-item adaptive measure (A-NUM). In a calibration study, 2 participant samples (n = 226 and 264 from Amazon’s Mechanical Turk [MTurk]) each responded to half of 84 novel numeracy items. We calibrated items using 2-parameter logistic item response theory (IRT) models. Based on item parameters, we developed the 3 new numeracy measures. In a subsequent validation study, 600 MTurk participants completed the new numeracy measures, the adaptive Berlin Numeracy Test, and the Weller Rasch-Based Numeracy Test, in randomized order. To establish predictive and convergent validities, participants also completed judgment and decision tasks, Raven’s progressive matrices, a vocabulary test, and demographics. Confirmatory factor analyses suggested that the 1-NUM, 4-NUM, and A-NUM load onto the same factor as existing measures. The NUM scales also showed similar association patterns to subjective numeracy and cognitive ability measures as established measures. Finally, they effectively predicted classic numeracy effects. In fact, based on power analyses, the A-NUM and 4-NUM appeared to confer more power to detect effects than existing measures. Thus, using IRT, we developed 3 brief numeracy measures, using novel items and without sacrificing construct scope. The measures can be downloaded as Qualtrics files (https://osf.io/pcegz/).
more »
« less
- Award ID(s):
- 2017651
- PAR ID:
- 10469962
- Publisher / Repository:
- Cambridge
- Date Published:
- Journal Name:
- Judgment and Decision Making
- Volume:
- 18
- ISSN:
- 1930-2975
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Visualization misinformation is a prevalent problem, and combating it requires understanding people’s ability to read, interpret, and reason about erroneous or potentially misleading visualizations, which lacks a reliable measurement: existing visualization literacy tests focus on well-formed visualizations. We systematically develop an assessment for this ability by: (1) developing a precise definition of misleaders (decisions made in the construction of visualizations that can lead to conclusions not supported by the data), (2) constructing initial test items using a design space of misleaders and chart types, (3) trying out the provisional test on 497 participants, and (4) analyzing the test tryout results and refining the items using Item Response Theory, qualitative analysis, a wrong-due-to-misleader score, and the content validity index. Our final bank of 45 items shows high reliability, and we provide item bank usage recommendations for future tests and different use cases. Related materials are available at: https://osf.io/pv67z/.more » « less
-
Park and Brannon (2013, https://doi.org/10.1177/0956797613482944) found that practicing non-symbolic approximate arithmetic increased performance on an objective numeracy task, specifically symbolic arithmetic. Manipulating objective numeracy would be useful for many researchers, particularly those who wish to investigate causal effects of objective numeracy on performance. Objective numeracy has been linked to performance in multiple areas, such as judgment-and-decision-making (JDM) competence, but most existing studies are correlational. Here, we expanded upon Park and Brannon’s method to experimentally manipulate objective numeracy and we investigated whether numeracy’s link with JDM performance was causal. Experimental participants drawn from a diverse internet sample trained on approximate-arithmetic tasks whereas active control participants trained on a spatial working-memory task. Numeracy training followed a 2 × 2 design: Experimental participants quickly estimated the sum of OR difference between presented numeric stimuli, using symbolic numbers (i.e., Arabic numbers) OR non-symbolic numeric stimuli (i.e., dot arrays). We partially replicated Park and Brannon’s findings: The numeracy training improved objective-numeracy performance more than control training, but this improvement was evidenced by performance on the Objective Numeracy Scale, not the symbolic arithmetic task. Subsequently, we found that experimental participants also perceived risks more consistently than active control participants, and this risk-consistency benefit was mediated by objective numeracy. These results provide the first known experimental evidence of a causal link between objective numeracy and the consistency of risk judgments.more » « less
-
null (Ed.)Background. Numeracy skills are important for medical decision making as lower numeracy is associated with misinterpreting statistical health risks. Math anxiety, characterized by negative emotions about numerical tasks, and lower subjective numeracy (i.e., self-assessments of numerical competence) are also associated with poor risk comprehension. Objective. To explore independent and mediated associations of math anxiety, numerical ability, and subjective numeracy with risk comprehension and to ascertain whether their associations are specific to the health domain. Methods. Objective numeracy was measured with a 14-item test. Math anxiety and subjective numeracy were assessed with self-report scales. Risk comprehension was measured with a 12-item test. In experiment 1, risk comprehension items were limited to scenarios in the health domain. In experiment 2, participants were randomly assigned to receive numerically equivalent risk comprehension items in either a health or nonhealth domain. Results. Linear regression analyses revealed that individuals with higher objective numeracy were more likely to respond correctly to the risk comprehension items, as were individuals with higher subjective numeracy. Higher math anxiety was associated with a lower likelihood of correct responding when controlling for objective numeracy but not when controlling for subjective numeracy. Mediation analyses indicated that math anxiety may undermine risk comprehension in 3 ways, including through 1) objective numeracy, 2) subjective numeracy, and 3) objective and subjective numeracy in serial, with subjective numeracy mediating the association between objective numeracy and risk comprehension. Findings did not differ by domain. Conclusions. Math anxiety, objective numeracy, and subjective numeracy are associated with risk comprehension through unique pathways. Education initiatives for improving health risk comprehension may be most effective if jointly aimed at tackling numerical ability as well as negative emotions and self-evaluations related to numeracy.more » « less
-
Item response theory (IRT) has become one of the most popular statistical models for psychometrics, a field of study concerned with the theory and techniques of psychological measurement. The IRT models are latent factor models tailored to the analysis, interpretation, and prediction of individuals’ behaviors in answering a set of measurement items that typically involve categorical response data. Many important questions of measurement are directly or indirectly answered through the use of IRT models, including scoring individuals’ test performances, validating a test scale, linking two tests, among others. This paper provides a review of item response theory, including its statistical framework and psychometric applications. We establish connections between item response theory and related topics in statistics, including empirical Bayes, nonparametric methods, matrix completion, regularized estimation, and sequential analysis. Possible future directions of IRT are discussed from the perspective of statistical learning.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1017/jdm.2023.17
