Automatic Text Simplification (ATS) uses natural language processing (NLP) to rewrite text with the goal of reducing its linguistic complexity while preserving as much of its original meaning as possible [24]. Researchers in the area of ATS have explored different approaches, which include: lexical approaches, which aim to replace individual complex words with simpler synonyms, e.g. [16]; syntactic approaches, which aim to rewrite longer chunks of text such as phrases or sentences and reduce their syntactic complexity, e.g. [7]; or approaches that incorporate both syntactic and lexical changes, e.g. [17,28]. While the research on improving the underlying tecnologies remains prevalent as this is an emerging experimental field, a parallel line of research in human computer interaction (HCI) and computing accessibility also investigates the potential applications of these technologies to provide reading assistance to different user groups, including people with disabilities.
Research has observed great diversity in the literacy skills of DHH adults. While many DHH adults can be strong readers, some studies have found, for instance, 4th grade reading levels among DHH high school graduates [25] and other studies suggest that over 17% of DHH adults can be classified as having "low literacy" [9]. Low literacy can pose particular challenges to individuals in the computing and IT fields, which often requires professionals to learn new technology skills, often through text-based content. Thus, our research focuses particularly on the use of ATS to provide reading assistance to Deaf and Hard-of-hearing (DHH) adults in the computing and information field.
Prior accessibility research on the use of ATS as reading assistance tools has focused on different user groups, including people with dyslexia (e.g. [22]), people with aphasia (e.g. [8]), people with autism (e.g. [27]), low-literacy readers (e.g. [26]), and DHH adults [13,14] or children (e.g. [12]). Most of this prior research focuses on evaluating whether there are benefits obtained from using different ATS technologies and approaches with these user groups (e.g. [20,23]). However, prior work in the context of DHH adults have only evaluated syntactic approaches to text simplification [13]. Considering that prior education research has suggested that DHH readers often face difficulty at both the lexical-and syntactic-level [15], we explore whether there are benefits obtained from other approaches to ATS (e.g. lexical approaches) when using them to provide reading assistance to DHH readers.
Recent work in NLP has advocated for incorporating human judgements of complexity into the complexity scoring systems of ATS, as well as focusing on particular text domains [16] (i.e. texts that focus on a particular topic such as medical texts [7]). However, while some accessibility work has investigated the incorporation of direct feedback from users during the simplification process [5], to the best of our knowledge no prior work has gathered judgements on a wide lexicon from users with a particular literacy profile, nor from a particular domain. Thus, in our work we gather judgements on a large lexicon of both a general (as described in [16]) and a computing vocabulary from DHH individuals to support the use of these judgements in the complexity scoring systems of ATS systems. Some research has also focused on evaluating the text simplification needs of particular user groups [27], as well as the linguistic needs (as defined by what aspects of the text increase their complexity) of particular groups [19,21]. While the latter has suggested that different populations possess different literacy needs and linguistic profiles, considerations of how a particular literacy profile may be better served by different ATS approaches are usually not reported. Further, the linguistic and text simplification needs of DHH adults in the context of ATS have not been explored. Thus, in our work, we explore the simplification needs and requirements, and preferences for different ATS approaches of DHH adults in the computing and IT field.
Finally, many of the experimental studies in prior work have made particular choices about how the users interact with their prototypes, including decisions about the platform (e.g. on a website [23] vs. a browser extension [5]), the level of user initiative for requesting simplifications (e.g. automatically replacing text [20] vs. allowing the user to request simplifications [5,26]), the placement of the simpler text (fully replacing the text [20] vs. providing the simplified text in a tooltip while preserving the original text [10,20]), and the visibility that a replacement has taken place (e.g. providing a background highlight to convey that a change has happened [5] vs. providing no visible indications [6,13,22]. However, while prior work has speculated that the choices of interaction parameters may be more important than even the choice of simplification strategy [21], little work has focused on users' preferences for different settings of these parameters and how those may affect their perspectives on the systems usability, acceptability, as well as the benefits obtained. Thus, in our work we incorporate participatory approaches for driving the decisions of the different interface parameters, as well as summative evaluations to investigate the influence of those decisions in users' perspectives of the systems.
Our goal is to investigate user-centered solutions for using ATS as reading-assistance tools for DHH individuals in the computing and IT field. As outlined in the previous section, this incorporates investigating their needs and preferences, evaluating whether there are benefits obtained from different approaches to ATS, gathering their judgements on the complexity of general and computing lexicons, as well as investigating their preferences for different interface parameters which would allow us to create a prototype to support a summative evaluation of the technology. Each of these steps incorporates its own challenges, and there are different methodologies that we employ for these different stages of our project.
1. Investigating needs and preferences. We conduct a survey and follow-up interviews to investigate DHH adults' reading experiences, how they currently overcome complicated text, as well as their perspectives and interests on the use of ATS as reading assistance tools. We also conduct qualitative research to investigate the social accessibility of these tools, which may inform other interface parameters that will be important to consider in addition to the ones outlined above. 2. Investigating interface parameters. We employ participatory methods to investigate users' preferences for the tuning of the different interface parameters, including the platform, user initiative, placement and visibility of the replacement text, as outlined above, as well as the permanence of the replacement text, displays of the system's confidence of the appropriateness of a replacement, and other parameters that may emerge as important from our qualitative user studies. 3. Evaluating ATS. Typically, prior work has evaluated ATS and their benefits for users by conducting experimental studies, employing a variety of both objective and subjective judgements such as comprehension questions or judgements of readability or understandability. However, considering that prior work has identified challenges in evaluating linguistic technologies with DHH adults given their diversity in literacy [4,11], we first need to conduct methodological research on how to evaluate these technologies with DHH adults to ensure we can trust the metrics employed. 4. Gathering word complexity judgements. We follow the methodology of [16] to gather judgements from DHH adults on a 15,000 word general vocabulary lexicon, as well as on a computing lexicon of similar length obtained from computing sources online. 5. Summative evaluation. Based on the results from all the previous items, we develop a prototype to conduct experimental and/or observational studies with DHH adults in the computing and IT fields to evaluate its usability and social accessibility, as well as the benefits in comprehension when reading or engaging in self-directed learning tasks obtained from the use ATS-based reading assistance.
I am finishing the third year of my PhD program, and have completed all required course work for my program. Over the past three years, I have also been able to conduct research advancing many of the items outlined in the previous section through projects that have been disseminated at the ASSETS and CHI conferences, which are described below. This section also outlines on-going and the plans for future work, which will inform the direction of my dissertation proposal.
We have completed a survey and interview study focusing on the reading experiences and interests of DHH adults with experience in the computing and IT field [1]. In that study, we found that participants read often online for work and academic purposes, and expressed strong interest in ATS-based reading assistance tools.
The results from that study also highlighted concerns from participants around the accuracy and social accessibility of these tools, as well as interface parameters that would be important to consider such as how much text is replaced at once when a user requests a simplification.
We also conducted a methodological study on how to evaluate the complexity of simplified texts among DHH adults with different levels of literacy [3]. In that study, we found that comprehension questions were effective to distinguish text complexity only among DHH adults with lower literacy skills, and only when they were written with low linguistic complexity. Only subjective judgements of readability were effective with DHH adults at all literacy levels in our study.
Finally, we conducted an experimental study in which we evaluated whether there are benefits obtained by DHH adults from lexical simplification approaches, as well as participants' perspectives on the autonomy of ATS tools (as defined by the level of user initiative for requesting simplifications, as well as the placement and visibility of the replacement text that the interface provides to the user) [2]. In that study, we found that while our comprehension questions employed in that study did not yield significant differences, participants' subjective judgements suggested they perceived benefits from using lexical simplification. Participants also reported being more likely to use systems that provide them with the autonomy to request simplifications on demand, as well as visibility as for what text has been replaced.
Based on the results from [1], we are conducting further qualitative research focusing specifically on the social accessibility of ATS-based reading assistance tools. More specifically, we are conducting an interview study with DHH adults with experience in the computing industry to learn about their values and aspirations at work, and how the use of ATS-based reading assistance tools may support or conflict those values and aspirations in the social context of the work environment.
Considering that ATS is still an emerging technology and that errors may be introduced in the simplification process, we are also currently conducting further methodological work in which we investigate how to evaluate two other aspects of the quality of the output of ATS, which are often the subject of ATS evaluation: meaning preservation and fluency (or grammatical correctness). We are following an approach similar to our previous study [3] to ensure the integrity of the metrics identified as effective in that previous study when semantic and grammatical errors are introduced.
Finally, our data collection of judgements from DHH adults on the general and computing lexicons is also ongoing.
In the future, we plan to conduct research employing participatory methods for the design of a working prototype based on users' preferences for the different design parameters. We are planning to follow a similar approach to that of [18]. To support this, we are working on a highly customizable prototype that allows tuning the different interface parameters identified above. However, we are still on the planning stages for this project and it would be great to receive feedback on how to best design this study during the Doctoral Consortium, especially on how to best incorporate participatory design approaches to investigate the preferred settings for different interface parameters.
Based on all the work outlined above, we hope to conduct our summative evaluation in the next year by deploying our working prototype in experimental and/or observational studies. While our prototype will be informed by our qualitative and participatory research, these summative studies will also be informed by our methodological research on how to best evaluate ATS technologies with DHH adults. In these studies, we hope to explore aspects such as the usability and social accessibility of ATS-based reading assistance tools, as well as their potential benefits in reading comprehension and self-directed learning tasks. This is also an area of future work that could benefit from feedback during the Doctoral Consortium, especially considering the potential challenges of conducting observational studies in the current context of the COVID-19 pandemic where the future of in-person human-subject studies still remains unknown
While the main contribution of this research project will be a user-centered working prototype for ATSbased reading assistance tools that can be used for future evaluations of ATS technologies in accessibility research with DHH adults in the computing and information field, there are also other contributions that emerge from the many sub-projects in this work. These include:
1. Requirements and interests of DHH individuals in the computing field, suggesting that they frequently read on electronic devices to learn about new topics for their work, and expressed strong interest in such tools, which motivate the need for further technical and design work on ATS-based reading assistance tools for this group. 2. Datasets of judgements of complexity from DHH adults on general and computing lexicons, which can be used to tune ATS technologies and investigate how incorporating data from specific user groups or specific domains may affect their effectiveness. 3. Frameworks for conducting methodological research on how to evaluate different aspects of the quality of ATS output, including complexity, grammatical correctness and meaning preservation, which can be used with other user groups to ensure the proper metrics are employed in user evaluations. 4. A framework for investigating the preferences of different interface parameters with other user groups or those working in other domains for developing user-centered prototypes.
The ASSETS Doctoral consortium would be a great opportunity to receive feedback on our future studies, as outlined in the future work section above. I am planning to present my dissertation proposal in December 2021, so this would be the perfect timing to discuss our plans and receive feedback, and incorporate that feedback into my dissertation proposal. I would also love to have the opportunity to widen my network within the ASSETS community, and share my experiences and contributions with other students and senior members of the community participating in the Doctoral Consortium. Finally, as someone who comes from a developing country, where careers in scientific research are not common, I am always looking for opportunities to interact with researchers at different levels of seniority across academia and industry to learn more about their paths and experiences as a way to inform my decisions for my own career choices. So, having the opportunity to attend this Doctoral Consortium and the general ASSETS conference would be instrumental as the time for making those career choices approaches.