An official website of the United States government
What are the challenges of turning data subjects into research participants—and how can we approach this task responsibly? In this paper, we develop a methodology for studying the lived experiences of people who are subject to automated scoring systems. Unlike most media technologies, automated scoring systems are designed to track and rate specific qualities of people without their active participation. Credit scoring, risk assessments, and predictive policing all operate obliquely in the background long before they come to matter. In doing so, they constitute a problem not only for those subject to these systems but also for researchers who try to study their experience. Specifically, we identify three challenges that are distinct to studying experiences of automated scoring: limited awareness, embeddedness, and ongoing inquiry. Starting from the observation that coming to terms with one's position as a data subject constitutes a form of learning in its own right, we propose a research strategy called critical companionship. Originally articulated in the context of nursing research, critical companionship invites us to accompany a data subject over time, paying critical attention to how the participant's and the researcher's inquiries complicate and constitute each other. We illustrate the strengths and limitations of this methodology with materials from a recent study we conducted about people's credit repair practices and sketch a set of sensibilities for studying contemporary scoring systems from the margins.
more »
« less
Scoring thermal limits in small insects using open-source, computer-assisted motion detection
ABSTRACT Scoring thermal tolerance traits live or with recorded video can be time consuming and susceptible to observer bias, and as with many physiological measurements, there can be trade-offs between accuracy and throughput. Recent studies show that automated particle tracking is a viable alternative to manually scoring videos, although some of the software options are proprietary and costly. In this study, we present a novel strategy for automated scoring of thermal tolerance videos by inferring motor activity with motion detection using an open-source Python command line application called DIME (detector of insect motion endpoint). We apply our strategy to both dynamic and static thermal tolerance assays, and our results indicate that DIME can accurately measure thermal acclimation responses, generally agrees with visual estimates of thermal limits, and can significantly increase throughput over manual methods.
more »
« less
- Award ID(s):
- 1826689
- PAR ID:
- 10477849
- Publisher / Repository:
- Journal of Experimental Biology
- Date Published:
- Journal Name:
- Journal of Experimental Biology
- Volume:
- 226
- Issue:
- 22
- ISSN:
- 0022-0949
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Models for automated scoring of content in educational applications continue to demonstrate improvements in human-machine agreement, but it remains to be demonstrated that the models achieve gains for the “right” reasons. For providing reliable scoring and feedback, both high accuracy and connecting scoring decisions to scoring rubrics are crucial. We provide a quantitative and qualitative analysis of automated scoring models for science explanations of middle school students in an online learning environment that leverages saliency maps to explore the reasons for individual model score predictions. Our analysis reveals that top-performing models can arrive at the same predictions for very different reasons, and that current model architectures have difficulty detecting ideas in student responses beyond keywords.more » « less
-
Video analysis tools such as Tracker are used to study mechanical motion captured by photography. One can also imagine a similar tool for tracking thermal motion captured by thermography. Since its introduction to physics education, thermal imaging has been used to visualize phenomena that are invisible to the naked eye and teach a variety of physics concepts across different educational settings. But thermal cameras are still scarce in schools. Hence, videos recorded using thermal cameras such as those featured in “YouTube Physics” are suggested as alternatives. The downside is that students do not have interaction opportunities beyond playing those videos.more » « less
-
Abstract Under climate change, ectotherms will likely face pressure to adapt to novel thermal environments by increasing their upper thermal tolerance and its plasticity, a measure of thermal acclimation. Ectotherm populations with high thermal tolerance are often less thermally plastic, a trade‐off hypothesized to result from (i) a phenotypic limit on thermal tolerance above which plasticity cannot further increase the trait, (ii) negative genetic correlation or (iii) fitness trade‐offs between the two traits. Whether each hypothesis causes negative associations between thermal tolerance and plasticity has implications for the evolution of each trait.We empirically tested the limit and trade‐off hypotheses by leveraging the experimental tractability and thermal biology of the intertidal copepodTigriopus californicus. Using populations from four latitudinally distributed sites in coastal California, six lines per population were reared under a laboratory common garden for two generations. Ninety‐six full sibling replicates (n = 4–5 per line) from a third generation were developmentally conditioned to 21.5 and 16.5°C until adulthood. We then measured the upper thermal tolerance and fecundity of sibships at each temperature.We detected a significant trade‐off in fecundity, a fitness corollary, between baseline thermal tolerance and its plasticity.Tigriopus californicuspopulations and genotypes with higher thermal tolerance were less thermally plastic. We detected negative directional selection on thermal plasticity under ambient temperature evidenced by reduced fecundity. These fitness costs of plasticity were significantly higher among thermally tolerant genotypes, consistent with the trade‐off hypothesis. This trade‐off was evident under ambient conditions, but not high temperature.Observed thermal plasticity and fecundity were best explained by a model incorporating both the limit and trade‐off hypotheses rather than models with parameters associated with one hypothesis. Effects of population and family on tolerance and plasticity negatively covaried, suggesting that a negative genetic correlation could not be ruled as contributing to negative associations between the traits. Our study provides a novel empirical test of the fitness trade‐off hypothesis that leverages a strong inference approach. We discuss our results' insights into how thermal adaptation may be constrained by physiological limits, genetic correlations, and fitness trade‐offs between thermal tolerance and its plasticity. Read the freePlain Language Summaryfor this article on the Journal blog.more » « less
-
ABSTRACT Automated scoring is a current hot topic in creativity research. However, most research has focused on the English language and popular verbal creative thinking tasks, such as the alternate uses task. Therefore, in this study, we present a large language model approach for automated scoring of a scientific creative thinking task that assesses divergent ideation in experimental tasks in the German language. Participants are required to generate alternative explanations for an empirical observation. This work analyzed a total of 13,423 unique responses. To predict human ratings of originality, we used XLM‐RoBERTa (Cross‐lingual Language Model‐RoBERTa), a large, multilingual model. The prediction model was trained on 9,400 responses. Results showed a strong correlation between model predictions and human ratings in a held‐out test set (n = 2,682;r = 0.80; CI‐95% [0.79, 0.81]). These promising findings underscore the potential of large language models for automated scoring of scientific creative thinking in the German language. We encourage researchers to further investigate automated scoring of other domain‐specific creative thinking tasks.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1242/jeb.246548
