More Like this

1. Psychophysiological Arousal in Young Children Who Stutter: An Interpretable AI Approach

   https://doi.org/10.1145/3550326  

   Sharma, Harshit; Xiao, Yi; Tumanova, Victoria; Salekin, Asif (September 2022, Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies)

   The presented first-of-its-kind study effectively identifies and visualizes the second-by-second pattern differences in the physiological arousal of preschool-age children who do stutter (CWS) and who do not stutter (CWNS) while speaking perceptually fluently in two challenging conditions: speaking in stressful situations and narration. The first condition may affect children's speech due to high arousal; the latter introduces linguistic, cognitive, and communicative demands on speakers. We collected physiological parameters data from 70 children in the two target conditions. First, we adopt a novel modality-wise multiple-instance-learning (MI-MIL) approach to classify CWS vs. CWNS in different conditions effectively. The evaluation of this classifier addresses four critical research questions that align with state-of-the-art speech science studies' interests. Later, we leverage SHAP classifier interpretations to visualize the salient, fine-grain, and temporal physiological parameters unique to CWS at the population/group-level and personalized-level. While group-level identification of distinct patterns would enhance our understanding of stuttering etiology and development, the personalized-level identification would enable remote, continuous, and real-time assessment of stuttering children's physiological arousal, which may lead to personalized, just-in-time interventions, resulting in an improvement in speech fluency. The presented MI-MIL approach is novel, generalizable to different domains, and real-time executable. Finally, comprehensive evaluations are done on multiple datasets, presented framework, and several baselines that identified notable insights on CWSs' physiological arousal during speech production. 

   more » « less
2. Self-supervised Speech Models for Word-Level Stuttered Speech Detection

   Shih, Yi-Jen; Gkalitsiou, Zoi; Dimakis, Alexandros G; Harwath, David (September 2024, IEEE SLT 2024)

   Clinical diagnosis of stuttering requires an assessment by a licensed speech-language pathologist. However, this process is time-consuming and requires clinicians with training and experience in stuttering and fluency disorders. Unfortunately, only a small percentage of speech-language pathologists report being comfortable working with individuals who stutter, which is inadequate to accommodate for the 80 million individuals who stutter worldwide. Developing machine learning models for detecting stuttered speech would enable universal and automated screening for stuttering, enabling speech pathologists to identify and follow up with patients who are most likely to be diagnosed with a stuttering speech disorder. Previous research in this area has predominantly focused on utterance-level detection, which is not sufficient for clinical settings where word-level annotation of stuttering is the norm. In this study, we curated a stuttered speech dataset with word-level annotations and introduced a word-level stuttering speech detection model leveraging self-supervised speech models. Our evaluation demonstrates that our model surpasses previous approaches in word-level stuttering speech detection. Additionally, we conducted an extensive ablation analysis of our method, providing insight into the most important aspects of adapting self-supervised speech models for stuttered speech detection. 

   more » « less
3. Our Collective Voices: The Social and Technical Values of a Grassroots Chinese Stuttered Speech Dataset

   https://doi.org/10.1145/3715275.3732179  

   Li, Jingjin; Li, Qisheng; Gong, Rong; Wang, Lezhi; Wu, Shaomei (June 2025, ACM)

   The lack of authentic stuttered speech data has significantly limited the development of stuttering friendly automatic speech recognition (ASR) models. In previous work, we collaborated with StammerTalk, a grassroots community of Chinese-speaking people who stutter (PWS), to collect the first stuttered speech dataset in Mandarin Chinese, containing 50 hours of conversational and command-recitation speech from 72 PWS. This work examines both the technical and social dimensions of the dataset. Through quantitative and qualitative analysis, as well as benchmarking and fine-tuning ASR models using the dataset, we demonstrate its technical value in capturing stuttered speech at an unprecedented scale and diversity – enabling better understanding and mitigation of fluency bias in ASR – and its social value in promoting self-advocacy and structural change for PWS in China. By foregrounding lived experiences of PWS in their own voices, we also see the potential of this dataset to normalize speech disfluencies and cultivate deeper empathy for stuttering within the AI research community. 

   more » « less
4. A chimpanzee by any other name: The contributions of utterance context and information density on word choice

   https://doi.org/10.1016/j.cognition.2022.105265  

   Jacobs, Cassandra L.; MacDonald, Maryellen C. (January 2023, Cognition)

   An important feature of language production is the flexibility of lexical selection; producers could refer to an animal as chimpanzee, chimp, ape, she, and so on. Thus, a key question for psycholinguistic research is how and why producers make the lexical selections that they do. Information theoretic approaches have argued that producers regulate the uncertainty of the utterance for comprehenders, for example using longer words like chimpanzee if their messages are likely to be misunderstood, and shorter ones like chimp when the message is easy to understand. In this work, we test for the relative contributions of the information theoretic approach and an approach more aligned with psycholinguistic models of language production. We examine the effect on lexical selection of whole utterance-level factors that we take as a proxy for register or style in message-driven production accounts. Using a modern machine learning-oriented approach, we show that for both naturalistic stimuli and real-world corpora, producers prefer words to be longer in systematically different contexts, independent of the specific message they are trying to convey. We do not find evidence for regulation of uncertainty, as in information theoretic approaches. We offer suggestions for modification of the standard psycholinguistic production approach that emphasizes the need for the field to specify how message formulation influences lexical choice in multiword utterances. 

   more » « less
5. Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback

   Whitehill, Jacob; LoCasale-Crouch, Jennifer (June 2024, Journal of Educational Data Mining)

   With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, we explore how Large Language Models (LLMs) can be used to estimate "Instructional Support" domain scores of the CLassroom Assessment Scoring System (CLASS), a widely used observation protocol. We design a machine learning architecture that uses either zero-shot prompting of Meta's Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers' speech (transcribed automatically using OpenAI's Whisper) for the presence of Instructional Support. Then, these utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score. Experiments on two CLASS-coded datasets of toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS Instructional Support estimation accuracy using the proposed method (Pearson R up to 0.48) approaches human inter-rater reliability (up to R=0.55); (2) LLMs generally yield slightly greater accuracy than BoW for this task, though the best models often combined features extracted from both LLM and BoW; and (3) for classifying individual utterances, there is still room for improvement of automated methods compared to human-level judgments. Finally, (4) we illustrate how the model's outputs can be visualized at the utterance level to provide teachers with explainable feedback on which utterances were most positively or negatively correlated with specific CLASS dimensions. 

   more » « less