More Like this

1. Interest and Requirements for Sound-Awareness Technologies Among Deaf and Hard-of-Hearing Users of Assistive Listening Devices

   https://doi.org/10.1007/978-3-030-49108-6_11  

   Yeung, Peter; Alonzo, Oliver; Huenerfauth, Matt (July 2020, Universal Access in Human-Computer Interaction. Applications and Practice. HCII 2020. Lecture Notes in Computer Science)

   Antona, M; Stephanidis, C (Ed.)

   Environmental sounds can provide important information about surrounding activity, yet recognizing sounds can be challenging for Deaf and Hard-of-Hearing (DHH) individuals. Prior work has examined the preferences of DHH users for various sound-awareness methods. However, these preferences have been observed to vary along some demographic factors. Thus, in this study we investigate the preferences of a specific group of DHH users: current assistive listening devices users. Through a survey of 38 participants, we investigated their challenges and requirements for sound-awareness applications, as well as which type of sounds and what aspects of the sounds are of importance to them. We found that users of assistive listening devices still often miss sounds and rely on other people to obtain information about them. Participants indicated that the importance of awareness of different types of sounds varied according to the environment and the form factor of the sound-awareness technology. Congruent with prior work, participants reported that the location and urgency of the sound were of importance, as well as the confidence of the technology in its identification of that sound. 

   more » « less
2. Introducing locus: a nime for immersive exocentric aural environments

   Sardana, Disha; Joo, Woohun; Bukvic, Ivica Ico; Earle, Gregory. (January 2019, New Interfaces for Musical Expression)

   Locus is a NIME designed specifically for an interactive, immersive high density loudspeaker array environment. The system is based on a pointing mechanism to interact with a sound scene comprising 128 speakers. Users can point anywhere to interact with the system, and the spatial interaction utilizes motion capture, so it does not require a screen. Instead it is completely controlled via hand gestures using a glove that is populated with motion-tracking markers. The main purpose of this system is to offer intuitive physical interaction with the perimeter based spatial sound sources. Further, its goal is to minimize user-worn technology and thereby enhance freedom of motion by utilizing environmental sensing devices, such as motion capture cameras or infrared sensors. The ensuing creativity enabling technology is applicable to a broad array of possible scenarios, from researching limits of human spatial hearing perception to facilitating learning and artistic performances, including dance. Below we describe our NIME design and implementation, its preliminary assessment, and offer a Unity-based toolkit to facilitate its broader deployment and adoption. 

   more » « less
3. Visual to Sound: Generating Natural Sound for Videos in the Wild

   Zhou, Yipin; Wang, Zhaowen; Fang, Chen; Bui, Trung; Berg, Tamara L. (June 2018, IEEE Conference on Computer Vision and Pattern Recognition)

   As two of the five traditional human senses (sight, hearing, taste, smell, and touch), vision and sound are basic sources through which humans understand the world. Often correlated during natural events, these two modalities combine to jointly affect human perception. In this paper, we pose the task of generating sound given visual input. Such capabilities could help enable applications in virtual reality (generating sound for virtual scenes automatically) or provide additional accessibility to images or videos for people with visual impairments. As a first step in this direction, we apply learning-based methods to generate raw waveform samples given input video frames. We evaluate our models on a dataset of videos containing a variety of sounds (such as ambient sounds and sounds from people/animals). Our experiments show that the generated sounds are fairly realistic and have good temporal synchronization with the visual inputs. 

   more » « less
4. SoundWatch: deep learning for sound accessibility on smartwatches

   https://doi.org/10.1145/3531447  

   Jain, Dhruv; Ngo, Hung; Patel, Pratyush; Goodman, Steven; Nguyen, Khoa; Grossman-Kahn, Rachel; Findlater, Leah; Froehlich, Jon (June 2022, Communications of the ACM)

   Smartwatches have the potential to provide glanceable, always-available sound feedback to people who are deaf or hard of hearing (DHH). We present SoundWatch, a smartwatch-based deep learning application to sense, classify, and provide feedback about sounds occurring in the environment. To design SoundWatch, we first examined four low-resource sound classification models across four device architectures: watch-only, watch+phone, watch+phone+cloud, and watch+cloud. We found that the best model, VGG-lite, performed similar to the state of the art for nonportable devices although requiring substantially less memory (∼1/3 rd ) and that the watch+phone architecture provided the best balance among CPU, memory, network usage, and latency. Based on these results, we built and conducted a lab evaluation of our smartwatch app with eight DHH participants. We found support for our sound classification app but also uncovered concerns with misclassifications, latency, and privacy. 

   more » « less
5. Auditory environment diversity quantified using entropy from real-world hearing aid data

   https://doi.org/10.3389/fdgth.2023.1141917  

   Jorgensen, Erik; Xu, Jingjing; Chipara, Octav; Wu, Yu-Hsiang (April 2023, Frontiers in Digital Health)

   Introduction Using data collected from hearing aid users’ own hearing aids could improve the customization of hearing aid processing for different users based on the auditory environments they encounter in daily life. Prior studies characterizing hearing aid users’ auditory environments have focused on mean sound pressure levels and proportions of environments based on classifications. In this study, we extend these approaches by introducing entropy to quantify the diversity of auditory environments hearing aid users encounter. Materials and Methods Participants from 4 groups (younger listeners with normal hearing and older listeners with hearing loss from an urban or rural area) wore research hearing aids and completed ecological momentary assessments on a smartphone for 1 week. The smartphone was programmed to sample the processing state (input sound pressure level and environment classification) of the hearing aids every 10 min and deliver an ecological momentary assessment every 40 min. Entropy values for sound pressure levels, environment classifications, and ecological momentary assessment responses were calculated for each participant to quantify the diversity of auditory environments encountered over the course of the week. Entropy values between groups were compared. Group differences in entropy were compared to prior work reporting differences in mean sound pressure levels and proportions of environment classifications. Group differences in entropy measured objectively from the hearing aid data were also compared to differences in entropy measured from the self-report ecological momentary assessment data. Results Auditory environment diversity, quantified using entropy from the hearing aid data, was significantly higher for younger listeners than older listeners. Entropy measured using ecological momentary assessment was also significantly higher for younger listeners than older listeners. Discussion Using entropy, we show that younger listeners experience a greater diversity of auditory environments than older listeners. Alignment of group entropy differences with differences in sound pressure levels and hearing aid feature activation previously reported, along with alignment with ecological momentary response entropy, suggests that entropy is a valid and useful metric. We conclude that entropy is a simple and intuitive way to measure auditory environment diversity using hearing aid data. 

   more » « less