More Like this

1. Speech Perception Is Speech Learning

   https://doi.org/10.1177/09637214251318726  

   Holt, Lori_L (April 2025, Current Directions in Psychological Science)

   Speech conveys both linguistic messages and a wealth of social and identity information about a talker. This information arrives as complex variations across many acoustic dimensions. Ultimately, speech communication depends on experience within a language community to develop shared long-term knowledge of the mapping from acoustic patterns to the category distinctions that support word recognition, emotion evaluation, and talker identification. A great deal of research has focused on the learning involved in acquiring long-term knowledge to support speech categorization. Inadvertently, this focus may give the impression of a mature learning endpoint. Instead, there seems to be no firm line between perception and learning in speech. The contributions of acoustic dimensions are malleably reweighted continuously as a function of regularities evolving in short-term input. In this way, continuous learning across speech impacts the very nature of the mapping from sensory input to perceived category. This article presents a case study in understanding how incoming sensory input—and the learning that takes place across it—interacts with existing knowledge to drive predictions that tune the system to support future behavior. 

   more » « less
2. A neural speech decoding framework leveraging deep learning and speech synthesis

   https://doi.org/10.1038/s42256-024-00824-8  

   Chen, Xupeng; Wang, Ran; Khalilian-Gourtani, Amirhossein; Yu, Leyao; Dugan, Patricia; Friedman, Daniel; Doyle, Werner; Devinsky, Orrin; Wang, Yao; Flinker, Adeen (April 2024, Nature Machine Intelligence)

   Decoding human speech from neural signals is essential for brain–computer interface (BCI) technologies that aim to restore speech in populations with neurological deficits. However, it remains a highly challenging task, compounded by the scarce availability of neural signals with corresponding speech, data complexity and high dimensionality. Here we present a novel deep learning-based neural speech decoding framework that includes an ECoG decoder that translates electrocorticographic (ECoG) signals from the cortex into interpretable speech parameters and a novel differentiable speech synthesizer that maps speech parameters to spectrograms. We have developed a companion speech-to-speech auto-encoder consisting of a speech encoder and the same speech synthesizer to generate reference speech parameters to facilitate the ECoG decoder training. This framework generates natural-sounding speech and is highly reproducible across a cohort of 48 participants. Our experimental results show that our models can decode speech with high correlation, even when limited to only causal operations, which is necessary for adoption by real-time neural prostheses. Finally, we successfully decode speech in participants with either left or right hemisphere coverage, which could lead to speech prostheses in patients with deficits resulting from left hemisphere damage. 

   more » « less
3. Investigating the cortical tracking of speech and music with sung speech

   https://doi.org/10.21437/Interspeech.2023-1949  

   Cantisani, Giorgia; Chalehchaleh, Amirhossein; Di Liberto, Giovanni; Shamma, Shihab (August 2023, ISCA)
4. Gesture–speech physics in fluent speech and rhythmic upper limb movements

   https://doi.org/10.1111/nyas.14532  

   Pouw, Wim; Jonge‐Hoekstra, Lisette; Harrison, Steven J.; Paxton, Alexandra; Dixon, James A. (May 2021, Annals of the New York Academy of Sciences)

   null (Ed.)
5. VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

   https://doi.org/10.18653/v1/2024.acl-long.673  

   Peng, Puyuan; Huang, Po-Yao; Li, Shang-Wen; Mohamed, Abdelrahman; Harwath, David (August 2024, Association for Computational Linguistics)