An official website of the United States government
Learning to process speech in a foreign language involves learning new representations for mapping the auditory signal to linguistic structure. Behavioral experiments suggest that even listeners that are highly proficient in a non-native language experience interference from representations of their native language. However, much of the evidence for such interference comes from tasks that may inadvertently increase the salience of native language competitors. Here we tested for neural evidence of proficiency and native language interference in a naturalistic story listening task. We studied electroencephalography responses of 39 native speakers of Dutch (14 male) to an English short story, spoken by a native speaker of either American English or Dutch. We modeled brain responses with multivariate temporal response functions, using acoustic and language models. We found evidence for activation of Dutch language statistics when listening to English, but only when it was spoken with a Dutch accent. This suggests that a naturalistic, monolingual setting decreases the interference from native language representations, whereas an accent in the listener's own native language may increase native language interference, by increasing the salience of the native language and activating native language phonetic and lexical representations. Brain responses suggest that such interference stems from words from the native language competing with the foreign language in a single word recognition system, rather than being activated in a parallel lexicon. We further found that secondary acoustic representations of speech (after 200 ms latency) decreased with increasing proficiency. This may reflect improved acousticphonetic models in more proficient listeners. Significance StatementBehavioral experiments suggest that native language knowledge interferes with foreign language listening, but such effects may be sensitive to task manipulations, as tasks that increase metalinguistic awareness may also increase native language interference. This highlights the need for studying non-native speech processing using naturalistic tasks. We measured neural responses unobtrusively while participants listened for comprehension and characterized the influence of proficiency at multiple levels of representation. We found that salience of the native language, as manipulated through speaker accent, affected activation of native language representations: significant evidence for activation of native language (Dutch) categories was only obtained when the speaker had a Dutch accent, whereas no significant interference was found to a speaker with a native (American) accent.
more »
« less
Unveiling Performance Bias in ASR Systems: A Study on Gender, Age, Accent, and More
With the recent advancements in speech recognition, it is crucial to ensure these systems are free from performance biases against any speaker subgroups. This study examined the performance of twenty variants of seven Automatic Speech Recognition models across four datasets in English language: L2 Arctic, Speech Accent Archive, CORAAL, and SBCSAE. We employed Poisson regression and drop-in-deviance tests to identify which attributes significantly contribute to the Word Error Rate. Our analysis revealed biases related to attributes such as native language, location, occupation, and birthplace. Most systems did not exhibit bias related to factors like gender and age. Additionally, we conducted an experiment to detect bias related to "variant" (accent and dialect) by combining the CORAAL (African American Vernacular English (AAVE)) and SBCSAE (General American English (GAE)) datasets, aiming to identify the sources of any observed bias. We found that both speaker variability and dialectal difference contribute to observed bias for variant.
more »
« less
- Award ID(s):
- 2147350
- PAR ID:
- 10674697
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3503-6874-1
- Page Range / eLocation ID:
- 1 to 5
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract This article addresses bias in Spoken Language Systems (SLS) that involve both Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) and reports experiments to improve the performance of SLS for automated language and literacy‐related assessments with students who are under served in the U.S. educational system. We frame bias in SLS in terms of testing fairness and validity, stemming in part from the exclusion of sufficiently large training datasets in varieties of English other than General American English (GAE). We adopt an Interpretation/Use Argument approach to validity focused on clarity of constructs and scoring accuracy. While SLS use ASR to automatically transcribe students’ utterances, and apply NLP algorithms to ASR transcripts to measure students’ speech samples, it is well‐documented in studies with adults that ASR is typically more problematic for African American English (AAE) speakers than for other groups due to differences in prosody, pronunciation, word usage, and grammar. We utilized child speech and text corpora to improve algorithms that score oral task responses for child AAE speakers and, in some experiments, children with oral language and reading difficulties. Favorable results provide impetus and possible solutions for fair and inclusive assessments for diverse student groups in the future.more » « less
-
Fairness in speech processing systems is a critical challenge, especially regarding performance disparities based on speakers’ backgrounds. To help combat this problem, we are introducing FaiST (Fairness in Speech Technology), a novel speech dataset from American English speakers of various racial, ethnic, and national origin groups. The goal is to evaluate and mitigate possible biases across speech technologies using conversational speech from podcasts and interviews online. In FaiST’s current version, speakers self-identified as Asian American and African American, and future iterations will include other groups. White American speakers’ speech was extracted from VoxCeleb, and their demographic labels were obtained online. In addition to identifiers of race, ethnicity, and national origins, FaiST is also marked for the exact instances in the conversation where self-identifications occurred. We experimented with FaiST and found racial bias in eighteen Automatic Speech Recognition systems.more » « less
-
The rise of artificial intelligence has prompted increased scrutiny of systemic biases in automatic speech recognition technologies. One focal topic of discussion has been the degraded performance for speakers of African American and Southern U.S. English. This study aims to contribute to the research on bias in voice-AI by investigating speech recognition performance for Appalachian English, an often-stigmatized variety in American society. Participants were recruited from Southern Appalachia (Eastern Tennessee), with a non-Southern Appalachian (Central Pennsylvania) sample included as a reference group. The participants read aloud a version of the “Goldilocks and the Three Bears” fairy tale and the Rainbow Passage, and the recordings were processed using Dartmouth Linguistic Automation (DARLA). The authors conducted two sets of analyses on the vowel phonemes. The first analysis assessed DARLA's effectiveness in recognizing vowels. The system returned higher phoneme error rates for Southern Appalachian speech compared to the non-Southern dataset. Next, they conducted a detailed error analysis on the misrecognized input-output phoneme pairs. The results suggested dialect bias in the system, with 50.2% of the errors in the Southern dataset attributed to participation in the Southern Vowel Shift. These findings underscore the importance of integrating sociolectal variation into the acoustic model to mitigate dialect bias for currently underserved users.more » « less
-
We evaluate several publicly available off-the-shelf (commercial and research) automatic speech recognition (ASR) systems on dialogue agent-directed English speech from speakers with General American vs. non-American accents. Our results show that the performance of the ASR systems for non-American accents is considerably worse than for General American accents. Depending on the recognizer, the absolute difference in performance between General American accents and all non-American accents combined can vary approximately from 2% to 12%, with relative differences varying approximately between 16% and 49%. This drop in performance becomes even larger when we consider specific categories of non-American accents indicating a need for more diligent collection of and training on non-native English speaker data in order to narrow this performance gap. There are performance differences across ASR systems, and while the same general pattern holds, with more errors for non-American accents, there are some accents for which the best recognizer is different than in the overall case. We expect these results to be useful for dialogue system designers in developing more robust inclusive dialogue systems, and for ASR providers in taking into account performance requirements for different accents.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1109/ICASSP49660.2025.10887767
