skip to main content

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


  • NSF Public Access
  • Search Results
  • Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
Title: Methods for Evaluating the Fluency of Automatically Simplified Texts with Deaf and Hard-of-Hearing Adults at Various Literacy Levels
Research has revealed benefits and interest among Deaf and Hard-of-Hearing (DHH) adults in reading-assistance tools powered by Automatic Text Simplification (ATS), a technology whose development benefits from evaluations by specific user groups. While prior work has provided guidance for evaluating text complexity among DHH adults, researchers lack guidance for evaluating the fluency of automatically simplified texts, which may contain errors from the simplification process. Thus, we conduct methodological research on the effectiveness of metrics (including reading speed; comprehension questions; and subjective judgements of understandability, readability, grammaticality, and system performance) for evaluating texts controlled to be at different levels of fluency, when measured among DHH participants at different literacy levels. Reading speed and grammaticality judgements effectively distinguished fluency levels among participants across literacy levels. Readability and understandability judgements, however, only worked among participants with higher literacy. Our findings provide methodological guidance for designing ATS evaluations with DHH participants.  more » « less
Award ID(s):
1822747
PAR ID:
10335706
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22)
Page Range / eLocation ID:
1 to 10
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; ( , Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems)
    null (Ed.)
    Research has explored using Automatic Text Simplification for reading assistance, with prior work identifying benefits and interests from Deaf and Hard-of-Hearing (DHH) adults. While the evaluation of these technologies remains a crucial aspect of research in the area, researchers lack guidance in terms of how to evaluate text complexity with DHH readers. Thus, in this work we conduct methodological research to evaluate metrics identified from prior work (including reading speed, comprehension questions, and subjective judgements of understandability and readability) in terms of their effectiveness for evaluating texts modified to be at various complexity levels with DHH adults at different literacy levels. Subjective metrics and low-linguistic-complexity comprehension questions distinguished certain text complexity levels with participants with lower literacy. Among participants with higher literacy, only subjective judgements of text readability distinguished certain text complexity levels. For all metrics, participants with higher literacy scored higher or provided more positive subjective judgements overall. 
    more » « less
  2. ; ; ; ; ( , Proceedings of the Workshop on Text Simplification, Accessibility, and Readability (TSAR-2022))
    Research has explored the use of automatic text simplification (ATS), which consists of techniques to make text simpler to read, to provide reading assistance to Deaf and Hard-of-hearing (DHH) adults with various literacy levels. Prior work in this area has identified interest in and benefits from ATS-based reading assistance tools. However, no prior work on ATS has gathered judgements from DHH adults as to what constitutes complex text. Thus, following approaches in prior NLP work, this paper contributes new word-complexity judgements from 11 DHH adults on a dataset of 15,000 English words that had been previously annotated by L2 speakers, which we also augmented to include automatic annotations of linguistic characteristics of the words. Additionally, we conduct a supplementary analysis of the interaction effect between the linguistic characteristics of the words and the groups of annotators. This analysis highlights the importance of collecting judgements from DHH adults for training ATS systems, as it revealed statistically significant interaction effects for nearly all of the linguistic characteristics of the words. 
    more » « less
  3. ; ; ; ( , Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems)
    Automatic Text Simplification (ATS), which replaces text with simpler equivalents, is rapidly improving. While some research has examined ATS reading-assistance tools, little has examined preferences of adults who are deaf or hard-of-hearing (DHH), and none empirically evaluated lexical simplification technology (replacement of individual words) with these users. Prior research has revealed that U.S. DHH adults have lower reading literacy on average than their hearing peers, with unique characteristics to their literacy profile. We investigate whether DHH adults perceive a benefit from lexical simplification applied automatically or when users are provided with greater autonomy, with on-demand control and visibility as to which words are replaced. Formative interviews guided the design of an experimental study, in which DHH participants read English texts in their original form and with lexical simplification applied automatically or on-demand. Participants indicated that they perceived a benefit form lexical simplification, and they preferred a system with on-demand simplification. 
    more » « less
  4. ; ; ; ; ; ( , ACM Transactions on Accessible Computing)
    Automatic Text Simplification (ATS) software aims at automatically rewrite complex text to make it simpler to read. Prior research has explored the use of ATS as a reading assistance technology, identifying benefits from providing these technologies to different groups of users, including Deaf and Hard-of-hearing (DHH) adults. However, little work has investigated the interests and requirements of specific groups of potential users of this technology. Considering prior work establishing that computing professionals often need to read about new technologies in order to stay current in their profession, in this study, we investigated the reading experiences and interests of DHH individuals with work experience in the computing industry in ATS-based reading assistance tools, as well as their perspective on the social accessibility of those tools. Through a survey and two sets of interviews, we found that these users read relatively often, especially in support of their work, and were interested in tools to assist them with complicated texts; but misperceptions arising from public use of these tools may conflict with participants’ desired image in a professional context. This empirical contribution motivates further research into ATS-based reading assistance tools for these users, prioritizing which reading activities users are most interested in seeing the application of this technology, and highlighting design considerations for creating ATS tools for DHH adults, including considerations for social accessibility. 
    more » « less
  5. ; ; ; ( , The 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '20))
    null (Ed.)
    Automatic Text Simplification (ATS) software replaces text with simpler alternatives. While some prior research has explored its use as a reading assistance technology, including some empirical findings suggesting benefits for deploying this technology among particular groups of users, relatively little work has investigated the interest and requirements of specific groups of users of this technology. In this study, we investigated the interests of Deaf and Hard-of-Hearing (DHH) individuals in the computing industry in ATS-based reading assistance tools, motivated by prior work establishing that computing professionals often need to read about new technologies in order to stay current in their profession. Through a survey and follow-up interviews, we investigate these DHH individuals’ reading practices, current techniques for overcoming complicated text, and their interest in reading assistance tools for their work. Our results suggest that these users read relatively often, especially in support of their work, and they were interested in tools to assist them with complicated texts. This empirical contribution provides motivation for further research into ATS-based reading assistance tools for these users, prioritizing which reading activities users are most interested in seeing application of this technology, as well as some insights into design considerations for such tools. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email