More Like this

1. How Phonotactics Affect Multilingual and Zero-Shot ASR Performance

   https://doi.org/10.1109/ICASSP39728.2021.9414478  

   Feng, Siyuan; Zelasko, Piotr; Moro-Velazquez, Laureano; Abavisani, Ali; Hasegawa-Johnson, Mark; Scharenborg, Odette; Dehak, Najim (June 2021, ICASSP)

   null (Ed.)

   The idea of combining multiple languages’ recordings to train a single automatic speech recognition (ASR) model brings the promise of the emergence of universal speech representation. Recently, a Transformer encoder-decoder model has been shown to leverage multilingual data well in IPA transcriptions of languages presented during training. However, the representations it learned were not successful in zero-shot transfer to unseen languages. Because that model lacks an explicit factorization of the acoustic model (AM) and language model (LM), it is unclear to what degree the performance suffered from differences in pronunciation or the mismatch in phonotactics. To gain more insight into the factors limiting zero-shot ASR transfer, we replace the encoder-decoder with a hybrid ASR system consisting of a separate AM and LM. Then, we perform an extensive evaluation of monolingual, multilingual, and crosslingual (zero-shot) acoustic and language models on a set of 13 phonetically diverse languages. We show that the gain from modeling crosslingual phonotactics is limited, and imposing a too strong model can hurt the zero-shot transfer. Furthermore, we find that a multilingual LM hurts a multilingual ASR system’s performance, and retaining only the target language’s phonotactic data in LM training is preferable. 

   more » « less
2. Autosegmental Neural Nets: Should Phones and Tones be Synchronous or Asynchronous?

   https://doi.org/10.21437/Interspeech.2020-1834  

   Li, Jialu; Hasegawa-Johnson, Mark (October 2020, Interspeech)

   null (Ed.)

   Phones, the segmental units of the International Phonetic Al-phabet (IPA), are used for lexical distinctions in most human languages; Tones, the suprasegmental units of the IPA,are used in perhaps 70%. Many previous studies have explored cross-lingual adaptation of automatic speech recognition(ASR) phone models, but few have explored the multilingual and cross-lingual transfer of synchronization between phones and tones. In this paper, we test four Connectionist Temporal Classification (CTC)-based acoustic models, differing in the degree of synchrony they impose between phones and tones.Models are trained and tested multilingually in three languages,then adapted and tested cross-lingually in a fourth. Both synchronous and asynchronous models are effective in both multi-lingual and cross-lingual settings. Synchronous models achieve lower error rate in the joint phone+tone tier, but asynchronous training results in lower tone error rate. 

   more » « less
3. Multi-Source Cross-Lingual Model Transfer: Learning What to Share

   Chen, Xilun; Hassan Awadallah, Ahmed; Hassan, Hany; Wang, Wei; Cardie, Claire (January 2019, Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL))

   Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applicable to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by leveraging labeled data from other (source) languages. In this work, we focus on the multilingual transfer setting where training data in multiple source languages is leveraged to further boost target language performance. Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both language invariant and language-specific features at instance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target language and each individual source language1. This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource setting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classification and sequence tagging. 

   more » « less
4. Synthetic Data Augmentation for Improving Low-Resource ASR

   https://doi.org/10.1109/WNYIPW.2019.8923082  

   Thai, Bao; Jimerson, Robert; Arcoraci, Dominic; Prud'hommeaux, Emily; Ptucha, Raymond (October 2019, 2019 IEEE Western New York Image and Signal Processing Workshop (WNYISPW))

   Although the application of deep learning to automatic speech recognition (ASR) has resulted in dramatic reductions in word error rate for languages with abundant training data, ASR for languages with few resources has yet to benefit from deep learning to the same extent. In this paper, we investigate various methods of acoustic modeling and data augmentation with the goal of improving the accuracy of a deep learning ASR framework for a low-resource language with a high baseline word error rate. We compare several methods of generating synthetic acoustic training data via voice transformation and signal distortion, and we explore several strategies for integrating this data into the acoustic training pipeline. We evaluate our methods on an indigenous language of North America with minimal training resources. We show that training initially via transfer learning from an existing high-resource language acoustic model, refining weights using a heavily concentrated synthetic dataset, and finally fine-tuning to the target language using limited synthetic data reduces WER by 15% over just transfer learning using deep recurrent methods. Further, we show improvements over traditional frameworks by 19% using a similar multistage training with deep convolutional approaches. 

   more » « less
5. Pre-training on high-resource speech recognition improves low-resource speech-to-text translation

   Bansal, S.; Kamper, H.; Livescu, K.; Lopez, A.; Goldwater, S. (January 2019, NAACL 2019)

   We present a simple approach to improve direct speech-to-text translation (ST) when the source language is low-resource: we pre-train the model on a high-resource automatic speech recognition (ASR) task, and then fine-tune its parameters for ST. We demonstrate that our approach is effective by pre-training on 300 hours of English ASR data to improve SpanishEnglish ST from 10.8 to 20.2 BLEU when only 20 hours of Spanish-English ST training data are available. Through an ablation study, we find that the pre-trained encoder (acoustic model) accounts for most of the improvement, despite the fact that the shared language in these tasks is the target language text, not the source language audio. Applying this insight, we show that pre-training on ASR helps ST even when the ASR language differs from both source and target ST languages: pre-training on French ASR also improves Spanish-English ST. Finally, we show that the approach improves performance on a true low-resource task: pre-training on a combination of English ASR and French ASR improves Mboshi-French ST, where only 4 hours of data are available, from 3.5 to 7.1 BLEU. 

   more » « less