An official website of the United States government
A variety of systems have been proposed to assist users in detecting machine learning (ML) fairness issues. These systems approach bias reduction from a number of perspectives, including recommender systems, exploratory tools, and dashboards. In this paper, we seek to inform the design of these systems by examining how individuals make sense of fairness issues as they use different de-biasing affordances. In particular, we consider the tension between de-biasing recommendations which are quick but may lack nuance and ”what-if” style exploration which is time consuming but may lead to deeper understanding and transferable insights. Using logs, think-aloud data, and semi-structured interviews we find that exploratory systems promote a rich pattern of hypothesis generation and testing, while recommendations deliver quick answers which satisfy participants at the cost of reduced information exposure. We highlight design requirements and trade-offs in the design of ML fairness systems to promote accurate and explainable assessments.
more »
« less
valuating the universal screeners for number sense: A validation study. I
The Universal Screeners for Number Sense (USNS) measure the construct of number sense, which has been characterized in numerous ways over the last 50 years. A purpose of this study is to provide validity evidence regarding the USNS assessments, which may support scholarship on numeracy and practical work in K-12 schools. Findings indicate that the USNS assessments may be used confidently to measure students’ growth in number sense.
more »
« less
- Award ID(s):
- 1920621
- PAR ID:
- 10519466
- Editor(s):
- Hammock, R; Cory, B
- Publisher / Repository:
- SSMA
- Date Published:
- Volume:
- 10
- Page Range / eLocation ID:
- 6-14
- Subject(s) / Keyword(s):
- number sense, formative assessment, validity, validation
- Format(s):
- Medium: X
- Location:
- Colorado Springs, CO
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Students who exhibit mature number sense make sense of numbers and operations, use reasoning to notice patterns, and flexibly choose effective problem-solving strategies (McIntosh et al., 1997, https://ro.ecu.edu.au/ecuworks/6819). Due to its dispositional nature, mature number sense is typically measured through in-depth interviews or tests of strategy usage. Yet, the lack of an efficient, rigorously developed measure has made it difficult to collect systematic, replicable evidence on students’ mature number sense. To address this, we developed a brief assessment of mature number sense. The present study provides additional convergent evidence of validity for this measure with US students in grades 3-8 (8–14 years old). We compared middle school (N = 40) and upper elementary school (N = 41) scores from the brief assessment with an established, time-intensive measure (Yang, 2019, https://doi.org/10.1007/s10649-018-9874-8) and an in-depth interview of student strategy usage (Markovits & Sowder, 1994, https://doi.org/10.2307/749290). We found strong correlations (r > 0.7) across all three measures, and this held even when controlling for students’ arithmetic scores (pr > 0.6). Researchers and educators can now use the brief assessment to investigate students’ mathematical thinking and advance knowledge of a key aspect of mathematical cognition.more » « less
-
This Brief Report presents an example of assessment validation using an argument-based approach. The instrument we developed is a Brief Assessment of Students’ Mature Number Sense, which measures a central goal in mathematics education. We chose to develop this assessment to provide an efficient way to measure the effect of instructional practices designed to improve students’ number sense. Using an argument-based framework, we first identify our proposed interpretations and uses of student scores. We then outline our argument with three claims that provide evidence connecting students’ responses on the assessment with its intended uses. Finally, we highlight why using argument-based validation benefits measure developers as well as the broader mathematics education community.more » « less
-
Abstract Number sense, the ability to decipher quantity, forms the foundation for mathematical cognition. How number sense emerges with learning is, however, not known. Here we use a biologically-inspired neural architecture comprising cortical layers V1, V2, V3, and intraparietal sulcus (IPS) to investigate how neural representations change with numerosity training. Learning dramatically reorganized neuronal tuning properties at both the single unit and population levels, resulting in the emergence of sharply-tuned representations of numerosity in the IPS layer. Ablation analysis revealed that spontaneous number neurons observed prior to learning were not critical to formation of number representations post-learning. Crucially, multidimensional scaling of population responses revealed the emergence of absolute and relative magnitude representations of quantity, including mid-point anchoring. These learnt representations may underlie changes from logarithmic to cyclic and linear mental number lines that are characteristic of number sense development in humans. Our findings elucidate mechanisms by which learning builds novel representations supporting number sense.more » « less
-
Visualization literacy is an essential skill for accurately interpreting data to inform critical decisions. Consequently, it is vital to understand the evolution of this ability and devise targeted interventions to enhance it, requiring concise and repeatable assessments of visualization literacy for individuals. However, current assessments, such as the Visualization Literacy Assessment Test ( vlat ), are time-consuming due to their fixed, lengthy format. To address this limitation, we develop two streamlined computerized adaptive tests ( cats ) for visualization literacy, a-vlat and a-calvi , which measure the same set of skills as their original versions in half the number of questions. Specifically, we (1) employ item response theory (IRT) and non-psychometric constraints to construct adaptive versions of the assessments, (2) finalize the configurations of adaptation through simulation, (3) refine the composition of test items of a-calvi via a qualitative study, and (4) demonstrate the test-retest reliability (ICC: 0.98 and 0.98) and convergent validity (correlation: 0.81 and 0.66) of both CATS via four online studies. We discuss practical recommendations for using our CATS and opportunities for further customization to leverage the full potential of adaptive assessments. All supplemental materials are available at https://osf.io/a6258/ .more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Proceeding:
- The DOI is not currently available.
