More Like this

1. Textless Speech-to-Speech Translation With Limited Parallel Data

   https://doi.org/10.18653/v1/2024.findings-emnlp.951  

   Diwan, Anuj; Srinivasan, Anirudh; Harwath, David; Choi, Eunsol (November 2024, Association for Computational Linguistics)
2. Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition

   https://doi.org/10.21437/Interspeech.2022-816  

   Ni, Junrui; Wang, Liming; Gao, Heting; Qian, Kaizhi; Zhang, Yang; Chang, Shiyu; Hasegawa-Johnson, Mark (September 2022, Proc. Interspeech 2022)

   An unsupervised text-to-speech synthesis (TTS) system learns to generate speech waveforms corresponding to any written sentence in a language by observing: 1) a collection of untranscribed speech waveforms in that language; 2) a collection of texts written in that language without access to any transcribed speech. Developing such a system can significantly improve the availability of speech technology to languages without a large amount of parallel speech and text data. This paper proposes an unsupervised TTS system based on an alignment module that outputs pseudo-text and another synthesis module that uses pseudo-text for training and real text for inference. Our unsupervised system can achieve comparable performance to the supervised system in seven languages with about 10-20 hours of speech each. A careful study on the effect of text units and vocoders has also been conducted to better understand what factors may affect unsupervised TTS performance. The samples generated by our models can be found at https://cactuswiththoughts.github.io/UnsupTTS-Demo, and our code can be found at https://github.com/lwang114/UnsupTTS. 

   more » « less
3. Transfer of statistical learning from passive speech perception to speech production

   https://doi.org/10.3758/s13423-023-02399-8  

   Murphy, Timothy_K; Nozari, Nazbanou; Holt, Lori_L (October 2023, Psychonomic Bulletin & Review)

   Abstract Communicating with a speaker with a different accent can affect one’s own speech. Despite the strength of evidence for perception-production transfer in speech, the nature of transfer has remained elusive, with variable results regarding the acoustic properties that transfer between speakers and the characteristics of the speakers who exhibit transfer. The current study investigates perception-production transfer through the lens of statistical learning across passive exposure to speech. Participants experienced a short sequence of acoustically variable minimal pair (beer/pier) utterances conveying either an accent or typical American English acoustics, categorized a perceptually ambiguous test stimulus, and then repeated the test stimulus aloud. In thecanonicalcondition, /b/–/p/ fundamental frequency (F0) and voice onset time (VOT) covaried according to typical English patterns. In thereversecondition, the F0xVOT relationship reversed to create an “accent” with speech input regularities atypical of American English. Replicating prior studies, F0 played less of a role in perceptual speech categorization in reverse compared with canonical statistical contexts. Critically, this down-weighting transferred to production, with systematic down-weighting of F0 in listeners’ own speech productions in reverse compared with canonical contexts that was robust across male and female participants. Thus, the mapping of acoustics to speech categories is rapidly adjusted by short-term statistical learning across passive listening and these adjustments transfer to influence listeners’ own speech productions. 

   more » « less
4. VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

   https://doi.org/10.18653/v1/2024.acl-long.673  

   Peng, Puyuan; Huang, Po-Yao; Li, Shang-Wen; Mohamed, Abdelrahman; Harwath, David (August 2024, Association for Computational Linguistics)
5. VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild

   Peng, P; Huang, P_Y; Li, S_W; Mohamed, A; Harwath, D (June 2024, https://doi.org/10.48550/arXiv.2403.16973)

   We introduce VoiceCraft, a token infilling neural codec language model, that achieves state-of-the-art performance on both speech editing and zero-shot text-to-speech (TTS) on audiobooks, internet videos, and podcasts. VoiceCraft employs a Transformer decoder architecture and introduces a token rearrangement procedure that combines causal masking and delayed stacking to enable generation within an existing sequence. On speech editing tasks, VoiceCraft produces edited speech that is nearly indistinguishable from unedited recordings in terms of naturalness, as evaluated by humans; for zero-shot TTS, our model outperforms prior SotA models including VALLE and the popular commercial model XTTS-v2. Crucially, the models are evaluated on challenging and realistic datasets, that consist of diverse accents, speaking styles, recording conditions, and background noise and music, and our model performs consistently well compared to other models and real recordings. In particular, for speech editing evaluation, we introduce a high quality, challenging, and realistic dataset named RealEdit. 

   more » « less