More Like this

1. Live Captions in Virtual Reality (VR)

   Pranav Pidathala, Dawson Franz (August 2023, Journal on Technology & Persons with Disabilities)

   Few VR applications and games implement captioning of speech and audio cues, which either inhibits or prevents access of their application by deaf or hard of hearing (DHH) users, new language learners, and other caption users. Additionally, little to no guidelines exist on how to implement live captioning on VR headsets and how it may differ from traditional television captioning. To help fill the void of information behind user preferences of different VR captioning styles, we conducted a study with eight DHH participants to test three caption movement behaviors (head-locked, lag, and appear- locked) while watching live-captioned, single-speaker presentations in VR. Participants answered a series of Likert scale and open-ended questions about their experience. Participants’ preferences were split, but most participants reported feeling comfortable with using live captions in VR and enjoyed the experience. When participants ranked the caption behaviors, there was almost an equal divide between the three types tested. IPQ results indicated each behavior had similar immersion ratings, however participants found head-locked and lag captions more user-friendly than appear-locked captions. We suggest that participants may vary in caption preference depending on how they use captions, and that providing opportunities for caption customization is best. 

   more » « less
2. Exploring the Design Space of Automatically Generated Emotive Captions for Deaf or Hard of Hearing Users

   https://doi.org/10.1145/3544549.3585880  

   Hassan, Saad; Ding, Yao; Kerure, Agneya Abhimanyu; Miller, Christi; Burnett, John; Biondo, Emily; Gilbert, Brenden (April 2023, Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (CHI EA '23))

   Caption text conveys salient auditory information to deaf or hard-of-hearing (DHH) viewers. However, the emotional information within the speech is not captured. We developed three emotive captioning schemas that map the output of audio-based emotion detection models to expressive caption text that can convey underlying emotions. The three schemas used typographic changes to the text, color changes, or both. Next, we designed a Unity framework to implement these schemas and used it to generate stimuli videos. In an experimental evaluation with 28 DHH viewers, we compared DHH viewers’ ability to understand emotions and their subjective judgments across the three captioning schemas. We found no significant difference in participants’ ability to understand the emotion based on the captions or their subjective preference ratings. Open-ended feedback revealed factors contributing to individual differences in preferences among the participants and challenges with automatically generated emotive captions that motivate future work. 

   more » « less
3. Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users

   https://doi.org/10.18653/v1/2022.ltedi-1.5  

   Amin, Akhter Al; Hassan, Saad; Alm, Cecilia; Huenerfauth, Matt (January 2022, Proceedings of the Second Workshop on Language Technology for Equality, Diversity and Inclusion)

   Deaf and hard of hearing individuals regularly rely on captioning while watching live TV. Live TV captioning is evaluated by regulatory agencies using various caption evaluation metrics. However, caption evaluation metrics are often not informed by preferences of DHH users or how meaningful the captions are. There is a need to construct caption evaluation metrics that take the relative importance of words in transcript into account. We conducted correlation analysis between two types of word embeddings and human-annotated labelled word-importance scores in existing corpus. We found that normalized contextualized word embeddings generated using BERT correlated better with manually annotated importance scores than word2vec-based word embeddings. We make available a pairing of word embeddings and their human-annotated importance scores. We also provide proof-of-concept utility by training word importance models, achieving an F1-score of 0.57 in the 6-class word importance classification task. 

   more » « less
4. What's in an ALT Tag? Exploring Caption Content Priorities through Collaborative Captioning

   https://doi.org/10.1145/3507659  

   Muehlbradt, Annika; Kane, Shaun K. (March 2022, ACM Transactions on Accessible Computing)

   Evaluating the quality of accessible image captions with human raters is difficult, as it may be difficult for a visually impaired user to know how comprehensive a caption is, whereas a sighted assistant may not know what information a user will need from a caption. To explore how image captioners and caption consumers assess caption content, we conducted a series of collaborative captioning sessions in which six pairs, consisting of a blind person and their sighted partner, worked together to discuss, create, and evaluate image captions. By making captioning a collaborative task, we were able to observe captioning strategies, to elicit questions and answers about image captions, and to explore blind users’ caption preferences. Our findings provide insight about the process of creating good captions and serve as a case study for cross-ability collaboration between blind and sighted people. 

   more » « less
5. Mixed Reality Speaker Identification as an Accessibility Tool for Deaf and Hard of Hearing Users

   https://doi.org/10.1145/3359996.3364720  

   Glasser, Abraham; Mason Riley, Edward; Weeks, Kaitlyn; Kushalnagar, Raja (November 2019, VRST '19: 25th ACM Symposium on Virtual Reality Software and Technology)

   People who are Deaf or Hard of Hearing (DHH) benefit from text captioning to understand audio, yet captions alone are often insufficient for the complex environment of a panel presentation, with rapid and unpredictable turn-taking among multiple speakers. It is challenging and tiring for DHH individuals to view captioned panel presentations, leading to feelings of misunderstanding and exclusion. In this work, we investigate the potential of Mixed Reality (MR) head-mounted displays for providing captioning with visual cues to indicate which person on the panel is speaking. For consistency in our experimental study, we simulate a panel presentation in virtual reality (VR) with various types of MR visual cues; in a study with 18 DHH participants, visual cues made it easier to identify speakers. 

   more » « less