More Like this

1. Unsupervised Text-to-Speech Synthesis by Unsupervised Automatic Speech Recognition

   https://doi.org/10.21437/Interspeech.2022-816  

   Ni, Junrui; Wang, Liming; Gao, Heting; Qian, Kaizhi; Zhang, Yang; Chang, Shiyu; Hasegawa-Johnson, Mark (September 2022, Proc. Interspeech 2022)

   An unsupervised text-to-speech synthesis (TTS) system learns to generate speech waveforms corresponding to any written sentence in a language by observing: 1) a collection of untranscribed speech waveforms in that language; 2) a collection of texts written in that language without access to any transcribed speech. Developing such a system can significantly improve the availability of speech technology to languages without a large amount of parallel speech and text data. This paper proposes an unsupervised TTS system based on an alignment module that outputs pseudo-text and another synthesis module that uses pseudo-text for training and real text for inference. Our unsupervised system can achieve comparable performance to the supervised system in seven languages with about 10-20 hours of speech each. A careful study on the effect of text units and vocoders has also been conducted to better understand what factors may affect unsupervised TTS performance. The samples generated by our models can be found at https://cactuswiththoughts.github.io/UnsupTTS-Demo, and our code can be found at https://github.com/lwang114/UnsupTTS. 

   more » « less
2. Variability of Sea Level and Upper-Ocean Heat Content in the Indian Ocean: Effects of Subtropical Indian Ocean Dipole and ENSO

   https://doi.org/10.1175/JCLI-D-19-0167.1  

   Zhang, Lei; Han, Weiqing; Li, Yuanlong; Lovenduski, Nicole S. (September 2019, Journal of Climate)

   Abstract In this study, the Indian Ocean upper-ocean variability associated with the subtropical Indian Ocean dipole (SIOD) is investigated. We find that the SIOD is associated with a prominent southwest–northeast sea level anomaly (SLA) dipole over the western-central south Indian Ocean, with the north pole located in the Seychelles–Chagos thermocline ridge (SCTR) and the south pole at southeast of Madagascar, which is different from the distribution of the sea surface temperature anomaly (SSTA). While the thermocline depth and upper-ocean heat content anomalies mirror SLAs, the air–sea CO2 flux anomalies associated with SIOD are controlled by SSTA. In the SCTR region, the westward propagation of oceanic Rossby waves generated by anomalous winds over the eastern tropical Indian Ocean is the major cause for the SLAs, with cyclonic wind causing negative SLAs during positive SIOD (pSIOD). Local wind forcing is the primary driver for the SLAs southeast of Madagascar, with anticyclonic winds causing positive SLAs. Since the SIOD is correlated with ENSO, the relative roles of the SIOD and ENSO are examined. We find that while ENSO can induce significant SLAs in the SCTR region through an atmospheric bridge, it has negligible impact on the SLA to the southeast of Madagascar. By contrast, the SIOD with ENSO influence removed is associated with an opposite SLA in the SCTR and southeast of Madagascar, corresponding to the SLA dipole identified above. A new subtropical dipole mode index (SDMI) is proposed, which is uncorrelated with ENSO and thus better represents the pure SIOD effect. 

   more » « less
3. Improving Black-box Speech Recognition using Semantic Parsing

   Corona, R.; Thomason, J.; Mooney, R.J. (November 2017, Proceedings of the 8th International Joint Conference on Natural Language Processing (IJCNLP-17))

   Speech is a natural channel for human-computer interaction in robotics and consumer applications. Natural language understanding pipelines that start with speech can have trouble recovering from speech recognition errors. Black-box automatic speech recognition (ASR) systems, built for general purpose use, are unable to take advantage of in-domain language models that could otherwise ameliorate these errors. In this work, we present a method for re-ranking black-box ASR hypotheses using an in-domain language model and semantic parser trained for a particular task. Our re-ranking method significantly improves both transcription accuracy and semantic understanding over a state-of-the-art ASR’s vanilla output. 

   more » « less
4. Automatic recognition of second language speech-in-noise

   https://doi.org/10.1121/10.0024877  

   Kim, Seung-Eun; Chernyak, Bronya_R; Seleznova, Olga; Keshet, Joseph; Goldrick, Matthew; Bradlow, Ann_R (February 2024, JASA Express Letters)

   Measuring how well human listeners recognize speech under varying environmental conditions (speech intelligibility) is a challenge for theoretical, technological, and clinical approaches to speech communication. The current gold standard—human transcription—is time- and resource-intensive. Recent advances in automatic speech recognition (ASR) systems raise the possibility of automating intelligibility measurement. This study tested 4 state-of-the-art ASR systems with second language speech-in-noise and found that one, whisper, performed at or above human listener accuracy. However, the content of whisper's responses diverged substantially from human responses, especially at lower signal-to-noise ratios, suggesting both opportunities and limitations for ASR--based speech intelligibility modeling. 

   more » « less
5. Prediction of Alzheimer's disease progression within 6 years using speech: A novel approach leveraging language models

   https://doi.org/10.1002/alz.13886  

   Amini, Samad; Hao, Boran; Yang, Jingmei; Karjadi, Cody; Kolachalama, Vijaya_B; Au, Rhoda; Paschalidis, Ioannis_C (June 2024, Alzheimer's & Dementia)

   Abstract INTRODUCTIONIdentification of individuals with mild cognitive impairment (MCI) who are at risk of developing Alzheimer's disease (AD) is crucial for early intervention and selection of clinical trials. METHODSWe applied natural language processing techniques along with machine learning methods to develop a method for automated prediction of progression to AD within 6 years using speech. The study design was evaluated on the neuropsychological test interviews ofn = 166 participants from the Framingham Heart Study, comprising 90 progressive MCI and 76 stable MCI cases. RESULTSOur best models, which used features generated from speech data, as well as age, sex, and education level, achieved an accuracy of 78.5% and a sensitivity of 81.1% to predict MCI‐to‐AD progression within 6 years. DISCUSSIONThe proposed method offers a fully automated procedure, providing an opportunity to develop an inexpensive, broadly accessible, and easy‐to‐administer screening tool for MCI‐to‐AD progression prediction, facilitating development of remote assessment. HighlightsVoice recordings from neuropsychological exams coupled with basic demographics can lead to strong predictive models of progression to dementia from mild cognitive impairment.The study leveraged AI methods for speech recognition and processed the resulting text using language models.The developed AI‐powered pipeline can lead to fully automated assessment that could enable remote and cost‐effective screening and prognosis for Alzehimer's disease. 

   more » « less