More Like this

1. Guiding Students to Investigate What Google Speech Recognition Knowsabout Language

   Touretzky, D. S.; Gardner-McCune, C.. (February 2023, Thirteenth AAAI Symposium on Educational Advances in Artificial Intelligence .)

   Children of all ages interact with speech recognition systems but are largely unaware of how they work. Teaching K-12 students to investigate how these systems employ phonolog- ical, syntactic, semantic, and cultural knowledge to resolve ambiguities in the audio signal can provide them a window on complex AI decision-making and also help them appreciate the richness and complexity of human language. We describe a browser-based tool for exploring the Google Web Speech API and a series of experiments students can engage in to measure what the service knows about language and the types of biases it exhibits. Middle school students taking an introductory AI elective were able to use the tool to explore Google’s knowledge of homophones and its ability to exploit context to disambiguate them. Older students could potentially conduct more comprehensive investigations, which we lay out here. This approach to investigating the power and limitations of speech technology through carefully designed experiments can also be applied to other AI application areas, such as face detection, object recognition, machine translation, or question answering. 

   more » « less
2. Neural network vs. HMM speech recognition systems as models of human cross-linguistic phonetic perception

   Schatz, Thomas; Feldman, Naomi H. (September 2018, Proceedings of the Conference on Cognitive Computational Neuroscience)

   The way listeners perceive speech sounds is largely determined by the language(s) they were exposed to as a child. For example, native speakers of Japanese have a hard time discriminating between American English /ɹ/ and /l/, a phonetic contrast that has no equivalent in Japanese. Such effects are typically attributed to knowledge of sounds in the native language, but quantitative models of how these effects arise from linguistic knowledge are lacking. One possible source for such models is Automatic Speech Recognition (ASR) technology. We implement models based on two types of systems from the ASR literature—hidden Markov models (HMMs) and the more recent, and more accurate, neural network systems—and ask whether, in addition to showing better performance, the neural network systems also provide better models of human perception. We find that while both types of systems can account for Japanese natives’ difficulty with American English /ɹ/ and /l/, only the neural network system successfully accounts for Japanese natives’ facility with Japanese vowel length contrasts. Our work provides a new example, in the domain of speech perception, of an often observed correlation between task performance and similarity to human behavior. 

   more » « less
3. Automatic Screening for Children with Speech Disorder Using Automatic Speech Recognition: Opportunities and Challenges

   https://doi.org/10.1609/aaaiss.v4i1.31807  

   Liu, Dancheng; Yang, Jason; Albrecht-Buehler, Ishan; Qin, Helen; Li, Sophie; Hu, Yuting; Nassereldine, Amir; Xiong, Jinjun (November 2024, Proceedings of the AAAI Symposium Series)

   Speech is a fundamental aspect of human life, crucial not only for communication but also for cognitive, social, and academic development. Children with speech disorders (SD) face significant challenges that, if unaddressed, can result in lasting negative impacts. Traditionally, speech and language assessments (SLA) have been conducted by skilled speech-language pathologists (SLPs), but there is a growing need for efficient and scalable SLA methods powered by artificial intelligence. This position paper presents a survey of existing techniques suitable for automating SLA pipelines, with an emphasis on adapting automatic speech recognition (ASR) models for children’s speech, an overview of current SLAs and their automated counterparts to demonstrate the feasibility of AI-enhanced SLA pipelines, and a discussion of practical considerations, including accessibility and privacy concerns, associated with the deployment of AI-powered SLAs. 

   more » « less
4. SAI: AI-Enabled Speech Assistant Interface for Science Gateways in HPC

   Pouya Kousha, Arpan Jain (May 2023, International Conference on High Performance Computing)

   High-Performance Computing (HPC) is increasingly being used in traditional scientific domains as well as emerging areas like Deep Learning (DL). This has led to a diverse set of professionals who interact with state-of-the-art HPC systems. The deployment of Science Gateways for HPC systems like Open On-Demand has a significant positive impact on these users in migrating their workflows to HPC systems. Although computing capabilities are ubiquitously available (as on-premises or in the cloud HPC infrastructure), significant effort and expertise are required to use them effectively. This is particularly challenging for domain scientists and other users whose primary expertise lies outside of computer science. In this paper, we seek to minimize the steep learning curve and associated complexities of using state-of-the-art high-performance systems by creating SAI: an AI-Enabled Speech Assistant Interface for Science Gateways in High Performance Computing. We use state-of-the-art AI models for speech and text and fine-tune them for the HPC arena by retraining them on a new HPC dataset we create. We use ontologies and knowledge graphs to capture the complex relationships between various components of the HPC ecosystem. We finally show how one can integrate and deploy SAI in Open OnDemand and evaluate its functionality and performance on real HPC systems. To the best of our knowledge, this is the first effort aimed at designing and developing an AI-powered speech-assisted interface for science gateways in HPC. 

   more » « less
5. Human and automated assessments of children's pronunciation

   https://doi.org/10.31274/psllt.18696  

   Novotny, Mackenzie; Richie, Carolyn; Levis, John M; Chukharev, Evgeny (June 2025, Proceedings of the annual Pronunciation in Second Language Learning and Teaching Conference)

   Patterns, types, and causes of errors in children’s pronunciation can be more variable than in adults’ speech. In school settings, different specialists work with children depending on their needs, including speech-language pathology (SLP) professionals and English as a second language (ESL) teachers. Because children’s speech is so variable, it is often difficult to identify which specialist is better suited to address a child’s needs. Computers excel at pattern recognition and can be quickly trained to identify a wide array of pronunciation issues, making them strong candidates to help with the difficult problem of identifying the appropriate specialist. As part of a larger project to create an automated pronunciation diagnostic tool to help identify which specialist a child may need, we created a pronunciation test for children between 5 and 7 years old. We recorded 26 children with a variety of language backgrounds and SLP needs and then compared automatic evaluations of their pronunciation to human evaluations. While the human evaluations showed high agreement, the automatic mispronunciation detection (MPD) system agreed on less than 50% of phonemes overall. However, the MPD showed consistent, albeit low, agreement across four subgroups of participants with no clear biases. Due to this performance, we recommend further research on children’s pronunciation and on specialized MPD systems that account for their unique speech characteristics and developmental patterns. 

   more » « less