Training acoustic models with sequentially incoming data –
while both leveraging new data and avoiding the forgetting effect
– is an essential obstacle to achieving human intelligence
level in speech recognition. An obvious approach to leverage
data from a new domain (e.g., new accented speech) is to first
generate a comprehensive dataset of all domains, by combining
all available data, and then use this dataset to retrain the
acoustic models. However, as the amount of training data
grows, storing and retraining on such a large-scale dataset
becomes practically impossible. To deal with this problem,
in this study, we study several domain expansion techniques
which exploit only the data of the new domain to build a
stronger model for all domains. These techniques are aimed at
learning the new domain with a minimal forgetting effect (i.e.,
they maintain original model performance). These techniques
modify the adaptation procedure by imposing new constraints
including (1) weight constraint adaptation (WCA): keeping
the model parameters close to the original model parameters;
(2) elastic weight consolidation (EWC): slowing down
training for parameters that are important for previously established
domains; (3) soft KL-divergence (SKLD): restricting
the KL-divergence between the original and the adapted
model output distributions; and (4) hybrid SKLD-EWC: incorporating
both SKLD and EWC constraints. We evaluate
these techniques in an accent adaptation task in which we
adapt a deep neural network (DNN) acoustic model trained
with native English to three different English accents: Australian,
Hispanic, and Indian. The experimental results show
that SKLD significantly outperforms EWC, and EWC works
better than WCA. The hybrid SKLD-EWC technique results
in the best overall performance.
more »
« less
Domain Expansion for End-to-End Speech Recognition: Applications for Accent/Dialect Speech
Training Automatic Speech Recognition (ASR) systems with sequentially incoming data from alternate domains is an essential milestone in order to reach human intelligibility level in speech recognition. The main challenge of sequential learning is that current adaptation techniques result in significant performance degradation for previously-seen domains.To mitigate the catastrophic forgetting problem, this study proposes effective domain expansion techniques for two scenarios: 1)where only new domain data is available, and 2) where both prior and new domain data are available. We examine the efficacy of the approaches through experiments on adapting a model trained with native English to different English accents. For the first scenario, we study several existing and proposed regularization-based approaches to mitigate performance loss of initial data.The experiments demonstrate the superior performanceo four proposed Soft KL-Divergence(SKLD)-Model Averaging (MA) approach. In this approach, SKLD first alleviates the forgetting problem during adaptation; next, MA makes the final efficient compromise between the two domains by averaging parameters of the initial and adapted models. For the second scenario, we explore several rehearsal-based approaches, which leverage initial data to maintain the original model performance.We propose Gradient Averaging (GA) as well as an approach which operates by averaging gradients computed for both initial and new domains. Experiments demonstrate that GA outperforms retraining and specifically designed continual learning approaches, such as Averaged Gradient Episodic Memory (AGEM). Moreover, GA significantly improves computational costs over the complete retraining approach.
more »
« less
- Award ID(s):
- 1918032
- NSF-PAR ID:
- 10478761
- Publisher / Repository:
- IEEE
- Date Published:
- Journal Name:
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Volume:
- 31
- ISSN:
- 2329-9290
- Page Range / eLocation ID:
- 762 - 774
- Subject(s) / Keyword(s):
- ["Accented speech","continuallearning","domain expansion","end-to-end systems","model adaptation","speech recognition"]
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Modeling cross-lingual speech emotion recognition (SER) has become more prevalent because of its diverse applications. Existing studies have mostly focused on technical approaches that adapt the feature, domain, or label across languages, without considering in detail the similarities be- tween the languages. This study focuses on domain adaptation in cross-lingual scenarios using phonetic constraints. This work is framed in a twofold manner. First, we analyze emotion-specific phonetic commonality across languages by identifying common vowels that are useful for SER modeling. Second, we leverage these common vowels as an anchoring mechanism to facilitate cross-lingual SER. We consider American English and Taiwanese Mandarin as a case study to demonstrate the potential of our approach. This work uses two in-the-wild natural emotional speech corpora: MSP-Podcast (American English), and BIIC-Podcast (Taiwanese Mandarin). The proposed unsupervised cross-lingual SER model using these phonetical anchors outperforms the baselines with a 58.64% of unweighted average recall (UAR).more » « less
-
more » « less
Abstract The identity of dark matter has remained surprisingly elusive. While terrestrial experiments may be able to nail down a model, an alternative method is to identify dark matter based on astrophysical or cosmological signatures. A particularly sensitive approach is based on the unique signature of dark matter substructure in galaxy–galaxy strong lensing images. Machine-learning applications have been explored for extracting this signal. Because of the limited availability of high-quality strong lensing images, these approaches have exclusively relied on simulations. Due to the differences with the real instrumental data, machine-learning models trained on simulations are expected to lose accuracy when applied to real data. Here domain adaptation can serve as a crucial bridge between simulations and real data applications. In this work, we demonstrate the power of domain adaptation techniques applied to strong gravitational lensing data with dark matter substructure. We show with simulated data sets representative of Euclid and Hubble Space Telescope observations that domain adaptation can significantly mitigate the losses in the model performance when applied to new domains. Lastly, we find similar results utilizing domain adaptation for the problem of lens finding by adapting models trained on a simulated data set to one composed of real lensed and unlensed galaxies from the Hyper Suprime-Cam. This technique can help domain experts build and apply better machine-learning models for extracting useful information from the strong gravitational lensing data expected from the upcoming surveys.
-
Speech and language development are early indicators of overall analytical and learning ability in children. The preschool classroom is a rich language environment for monitoring and ensuring growth in young children by measuring their vocal interactions with teachers and classmates. Early childhood researchers are naturally interested in analyzing naturalistic vs. controlled lab recordings to measure both quality and quantity of such interactions. Unfortunately, present-day speech technologies are not capable of addressing the wide dynamic scenario of early childhood classroom settings. Due to the diversity of acoustic events/conditions in such daylong audio streams, automated speaker diarization technology would need to be advanced to address this challenging domain for segmenting audio as well as information extraction. This study investigates an alternate Deep Learning-based diarization solution for segmenting classroom interactions of 3-5 year old children with teachers. In this context, the focus on speech-type diarization which classifies speech segments as being either from adults or children partitioned across multiple classrooms. Our proposed ResNet model achieves a best F1-score of ∼78.0% on data from two classrooms, based on dev and test sets of each classroom. It is utilized with Automatic Speech Recognition-based resegmentation modules to perform child-adult diarization. Additionally, F1-scores are obtained for individual segments with corresponding speaker tags (e.g., adult vs. child), which provide knowledge for educators on child engagement through naturalistic communications. The study demonstrates the prospects of addressing educational assessment needs through communication audio stream analysis, while maintaining both security and privacy of all children and adults. The resulting child communication metrics have been used for broad-based feedback for teachers with the help of visualizations.more » « less
-
more » « less
Operational networks commonly rely on machine learning models for many tasks, including detecting anomalies, inferring application performance, and forecasting demand. Yet, model accuracy can degrade due to concept drift, whereby the relationship between the features and the target to be predicted changes. Mitigating concept drift is an essential part of operationalizing machine learning models in general, but is of particular importance in networking's highly dynamic deployment environments. In this paper, we first characterize concept drift in a large cellular network for a major metropolitan area in the United States. We find that concept drift occurs across many important key performance indicators (KPIs), independently of the model, training set size, and time interval---thus necessitating practical approaches to detect, explain, and mitigate it. We then show that frequent model retraining with newly available data is not sufficient to mitigate concept drift, and can even degrade model accuracy further. Finally, we develop a new methodology for concept drift mitigation, Local Error Approximation of Features (LEAF). LEAF works by detecting drift; explaining the features and time intervals that contribute the most to drift; and mitigates it using forgetting and over-sampling. We evaluate LEAF against industry-standard mitigation approaches (notably, periodic retraining) with more than four years of cellular KPI data. Our initial tests with a major cellular provider in the US show that LEAF consistently outperforms periodic and triggered retraining on complex, real-world data while reducing costly retraining operations.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1109/TASLP.2022.3233238
