An official website of the United States government
Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.
more »
« less
Stealth Literacy Assessments via Educational Games
Literacy assessment is essential for effective literacy instruction and training. However, traditional paper-based literacy assessments are typically decontextualized and may cause stress and anxiety for test takers. In contrast, serious games and game environments allow for the assessment of literacy in more authentic and engaging ways, which has some potential to increase the assessment’s validity and reliability. The primary objective of this study is to examine the feasibility of a novel approach for stealthily assessing literacy skills using games in an intelligent tutoring system (ITS) designed for reading comprehension strategy training. We investigated the degree to which learners’ game performance and enjoyment predicted their scores on standardized reading tests. Amazon Mechanical Turk participants (n = 211) played three games in iSTART and self-reported their level of game enjoyment after each game. Participants also completed the Gates–MacGinitie Reading Test (GMRT), which includes vocabulary knowledge and reading comprehension measures. The results indicated that participants’ performance in each game as well as the combined performance across all three games predicted their literacy skills. However, the relations between game enjoyment and literacy skills varied across games. These findings suggest the potential of leveraging serious games to assess students’ literacy skills and improve the adaptivity of game-based learning environments.
more »
« less
- Award ID(s):
- 1828010
- PAR ID:
- 10432764
- Date Published:
- Journal Name:
- Computers
- Volume:
- 12
- Issue:
- 7
- ISSN:
- 2073-431X
- Page Range / eLocation ID:
- 130
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Westenberg, Dave J (Ed.)ABSTRACT Integrating primary scientific literature into Science, Technology, Engineering, and Mathematics (STEM) curricula enhances critical thinking, scientific literacy, and communication skills but presents challenges due to complex terminology and data interpretation barriers. To address these challenges, a scaffolded journal club approach was implemented in a Cancer Biology course. The course utilized Hypothes.is web-based annotations, methods presentations, figure annotations, and structured discussions to promote active engagement with the literature. Additionally, integrated science communication assignments—including written, graphical, and video abstracts—provided diverse opportunities for students to develop scientific literacy. This structured approach is designed to facilitate comprehension, encourage proactive learning, and foster confidence in engaging with primary scientific literature. Student feedback highlighted improved ability to dissect research articles, enhanced presentation skills, and increased enjoyment of scientific reading. The journal club model and science communication assignments offer a replicable framework for enhancing primary scientific literature engagement across various STEM disciplines and educational levels.more » « less
-
The need for improvement of societal disaster resilience and response efforts was evident after the destruction caused by the 2017 Atlantic hurricane season. We present a novel conceptual framework for improving disaster resilience through the combination of serious games, geographic information systems (GIS), spatial thinking, and disaster resilience. Our framework is implemented via Project Lily Pad, a serious geogame based on our conceptual framework, serious game case studies, interviews and real-life experiences from 2017 Hurricane Harvey survivors in Dickinson, TX, and an immersive hurricane-induced flooding scenario. The game teaches a four-fold set of skills relevant to spatial thinking and disaster resilience, including reading a map, navigating an environment, coding verbal instructions, and determining best practices in a disaster situation. Results of evaluation of the four skills via Project Lily Pad through a “think aloud” study conducted by both emergency management novices and professionals revealed that the game encouraged players to think spatially, can help build awareness for disaster response scenarios, and has potential for real-life use by emergency management professionals. It can be concluded from our results that the combination of serious games, geographic information systems (GIS), spatial thinking, and disaster resilience, as implemented via Project Lily Pad and our evaluation results, demonstrated the wide range of possibilities for using serious geogames to improve disaster resilience spatial thinking and potentially save lives when disasters occur.more » « less
-
Abstract We know that reading involves coordination between textual characteristics and visual attention, but research linking eye movements during reading and comprehension assessed after reading is surprisingly limited, especially for reading long connected texts. We tested two competing possibilities: (a) the weak association hypothesis: Links between eye movements and comprehension are weak and short‐lived, versus (b) the strong association hypothesis: The two are robustly linked, even after a delay. Using a predictive modeling approach, we trained regression models to predict comprehension scores from global eye movement features, using participant‐level cross‐validation to ensure that the models generalize across participants. We used data from three studies in which readers (Ns = 104, 130, 147) answered multiple‐choice comprehension questions ~30 min after reading a 6,500‐word text, or after reading up to eight 1,000‐word texts. The models generated accurate predictions of participants' text comprehension scores (correlations between observed and predicted comprehension: 0.384, 0.362, 0.372,ps < .001), in line with the strong association hypothesis. We found that making more, but shorter fixations, consistently predicted comprehension across all studies. Furthermore, models trained on one study's data could successfully predict comprehension on the others, suggesting generalizability across studies. Collectively, these findings suggest that there is a robust link between eye movements and subsequent comprehension of a long connected text, thereby connecting theories of low‐level eye movements with those of higher order text processing during reading.more » « less
-
null (Ed.)Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3390/computers12070130
