An official website of the United States government
Conducting human-centered research by, with, and for the ASL-signing Deaf community, requires rethinking current human-computer interaction processes in order to meet their linguistic and cultural needs and expectations. This paper highlights some key considerations that emerged in our work creating an ASL-based questionnaire, and our recommendations for handling them.
more »
« less
This content will become publicly available on April 25, 2026
Conducting HCI Research with the Deaf Community in American Sign Language: Practices and Experiences
Our team of culturally Deaf ASL-signing and hearing non-signing HCI researchers conduct research with the Deaf community to create ASL resources. This case study summarizes reflections, learning, and challenges with HCI user study protocols based on our experience conducting five user studies with deaf ASL-signing participants. The case study offers considerations for researchers in this space related to conducting think-aloud protocols, interviews and surveys, getting informed consent, interpreter services and data analysis and storage. Our goal is to share the lessons we learned, and offer recommendations for future research in this area. Going beyond accommodations and accessibility, we hope these reflections contribute to a shift toward ASL-centric HCI research methodologies for working with the Deaf Community.
more »
« less
- Award ID(s):
- 1901026
- PAR ID:
- 10587830
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400713958
- Page Range / eLocation ID:
- 1 to 8
- Subject(s) / Keyword(s):
- American Sign Language ASL Research Methods Human-Computer Interaction
- Format(s):
- Medium: X
- Location:
- Yokohama Japan
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Many technologies for human-computer interaction have been designed for hearing individuals and depend upon vocalized speech, precluding users of American Sign Language (ASL) in the Deaf community from benefiting from these advancements. While great strides have been made in ASL recognition with video or wearable gloves, the use of video in homes has raised privacy concerns, while wearable gloves severely restrict movement and infringe on daily life. Methods: This paper proposes the use of RF sensors for HCI applications serving the Deaf community. A multi-frequency RF sensor network is used to acquire non-invasive, non-contact measurements of ASL signing irrespective of lighting conditions. The unique patterns of motion present in the RF data due to the micro-Doppler effect are revealed using time-frequency analysis with the Short-Time Fourier Transform. Linguistic properties of RF ASL data are investigated using machine learning (ML). Results: The information content, measured by fractal complexity, of ASL signing is shown to be greater than that of other upper body activities encountered in daily living. This can be used to differentiate daily activities from signing, while features from RF data show that imitation signing by non-signers is 99% differentiable from native ASL signing. Feature-level fusion of RF sensor network data is used to achieve 72.5% accuracy in classification of 20 native ASL signs. Implications: RF sensing can be used to study dynamic linguistic properties of ASL and design Deaf-centric smart environments for non-invasive, remote recognition of ASL. ML algorithms should be benchmarked on native, not imitation, ASL data.more » « less
-
Video components are a central element of user interfaces that deliver content in a signed language (SL), but the potential of video components extends beyond content accessibility. Sign language videos may be designed as user interface elements: layered with interactive features to create navigation cues, page headings, and menu options. To be effective for signing users, novel sign language video-rich interfaces require informed design choices across many parameters. To align with the specific needs and shared conventions of the Deaf community and other ASL-signers in this context, we present a user study involving deaf ASL-signers who interacted with an array of designs for sign language video elements. Their responses offer some insights into how the Deaf community may perceive and prefer video elements to be designed, positioned, and implemented to guide user experiences. Through a qualitative analysis, we take initial steps toward understanding deaf ASL-signers’ perceptions of a set of emerging design principles, paving the way for future SL-centric user interfaces containing customized video elements and layouts with primary consideration for signed language-related usage and requirements.more » « less
-
null (Ed.)Current research in the recognition of American Sign Language (ASL) has focused on perception using video or wearable gloves. However, deaf ASL users have expressed concern about the invasion of privacy with video, as well as the interference with daily activity and restrictions on movement presented by wearable gloves. In contrast, RF sensors can mitigate these issues as it is a non-contact ambient sensor that is effective in the dark and can penetrate clothes, while only recording speed and distance. Thus, this paper investigates RF sensing as an alternative sensing modality for ASL recognition to facilitate interactive devices and smart environments for the deaf and hard-of-hearing. In particular, the recognition of up to 20 ASL signs, sequential classification of signing mixed with daily activity, and detection of a trigger sign to initiate human-computer interaction (HCI) via RF sensors is presented. Results yield %91.3 ASL word-level classification accuracy, %92.3 sequential recognition accuracy, 0.93 trigger recognition rate.more » « less
-
Raynal, Ann M.; Ranney, Kenneth I. (Ed.)Most research in technologies for the Deaf community have focused on translation using either video or wearable devices. Sensor-augmented gloves have been reported to yield higher gesture recognition rates than camera-based systems; however, they cannot capture information expressed through head and body movement. Gloves are also intrusive and inhibit users in their pursuit of normal daily life, while cameras can raise concerns over privacy and are ineffective in the dark. In contrast, RF sensors are non-contact, non-invasive and do not reveal private information even if hacked. Although RF sensors are unable to measure facial expressions or hand shapes, which would be required for complete translation, this paper aims to exploit near real-time ASL recognition using RF sensors for the design of smart Deaf spaces. In this way, we hope to enable the Deaf community to benefit from advances in technologies that could generate tangible improvements in their quality of life. More specifically, this paper investigates near real-time implementation of machine learning and deep learning architectures for the purpose of sequential ASL signing recognition. We utilize a 60 GHz RF sensor which transmits a frequency modulation continuous wave (FMWC waveform). RF sensors can acquire a unique source of information that is inaccessible to optical or wearable devices: namely, a visual representation of the kinematic patterns of motion via the micro-Doppler signature. Micro-Doppler refers to frequency modulations that appear about the central Doppler shift, which are caused by rotational or vibrational motions that deviate from principle translational motion. In prior work, we showed that fractal complexity computed from RF data could be used to discriminate signing from daily activities and that RF data could reveal linguistic properties, such as coarticulation. We have also shown that machine learning can be used to discriminate with 99% accuracy the signing of native Deaf ASL users from that of copysigning (or imitation signing) by hearing individuals. Therefore, imitation signing data is not effective for directly training deep models. But, adversarial learning can be used to transform imitation signing to resemble native signing, or, alternatively, physics-aware generative models can be used to synthesize ASL micro-Doppler signatures for training deep neural networks. With such approaches, we have achieved over 90% recognition accuracy of 20 ASL signs. In natural environments, however, near real-time implementations of classification algorithms are required, as well as an ability to process data streams in a continuous and sequential fashion. In this work, we focus on extensions of our prior work towards this aim, and compare the efficacy of various approaches for embedding deep neural networks (DNNs) on platforms such as a Raspberry Pi or Jetson board. We examine methods for optimizing the size and computational complexity of DNNs for embedded micro-Doppler analysis, methods for network compression, and their resulting sequential ASL recognition performance.more » « less
