More Like this

1. Child vs Adult Speaker Diarization of naturalistic audio recordings in preschool environment using Deep Neural Networks

   Kothalkar, P.; Hansen, J.H.L.; Buzhardt, J.; Irvin, D.; Rous, B. (March 2021, ASEE 2021 Gulf-Southwest Annual Conference)

   null (Ed.)

   Speech and language development in children is crucial for ensuring optimal outcomes in their long term development and life-long educational journey. A child’s vocabulary size at the time of kindergarten entry is an early indicator of learning to read and potential long-term success in school. The preschool classroom is thus a promising venue for monitoring growth in young children by measuring their interactions with teachers and classmates. Automatic Speech Recognition (ASR) technologies provide the ability for ‘Early Childhood’ researchers for automatically analyzing naturalistic recordings in these settings. For this purpose, data are collected in a high-quality childcare center in the United States using Language Environment Analysis (LENA) devices worn by the preschool children. A preliminary task for ASR of daylong audio recordings would involve diarization, i.e., segmenting speech into smaller parts for identifying ‘who spoke when.’ This study investigates a Deep Learning-based diarization system for classroom interactions of 3-5-year-old children. However, the focus is on ’speaker group’ diarization, which includes classifying speech segments as being from adults or children from across multiple classrooms. SincNet based diarization systems achieve utterance level Diarization Error Rate of 19.1%. Utterance level speaker group confusion matrices also show promising, balanced results. These diarization systems have potential applications in developing metrics for adult-to-child or child-to-child rapid conversational turns in a naturalistic noisy early childhood setting. Such technical advancements will also help teachers better and more efficiently quantify and understand their interactions with children, make changes as needed, and monitor the impact of those changes. 

   more » « less
2. Measuring Frequency of Child-directed WH-Question Words for Alternate Preschool Locations using Speech Recognition and Location Tracking Technologies

   https://doi.org/10.1145/3461615.3485440  

   Kothalkar, P.; Datla, S.; Dutta, S.; Hansen, J.H.L.; Seven, Y.; Irvin, D.; Buzhardt J. (October 2021, ACM ICMI - WoCBU-2021: Workshop on Bridging Social Sciences and AI for Understanding Child Behavior)

   Speech and language development in children are crucial for ensuring effective skills in their long-term learning ability. A child’s vocabulary size at the time of entry into kindergarten is an early indicator of their learning ability to read and potential long-term success in school. The preschool classroom is thus a promising venue for assessing growth in young children by measuring their interactions with teachers as well as classmates. However, to date limited studies have explored such naturalistic audio communications. Automatic Speech Recognition (ASR) technologies provide an opportunity for ’Early Childhood’ researchers to obtain knowledge through automatic analysis of naturalistic classroom recordings in measuring such interactions. For this purpose, 208 hours of audio recordings across 48 daylong sessions are collected in a childcare learning center in the United States using Language Environment Analysis (LENA) devices worn by the preschool children. Approximately 29 hours of adult speech and 26 hours of child speech is segmented using manual transcriptions provided by CRSS transcription team. Traditional as well as End-to-End ASR models are trained on adult/child speech data subset. Factorized Time Delay Neural Network provides a best Word-Error-Rate (WER) of 35.05% on the adult subset of the test set. End-to-End transformer models achieve 63.5% WER on the child subset of the test data. Next, bar plots demonstrating the frequency of WH-question words in Science vs. Reading activity areas of the preschool are presented for sessions in the test set. It is suggested that learning spaces could be configured to encourage greater adult-child conversational engagement given such speech/audio assessment strategies. 

   more » « less
3. Towards the development of personalized learning companion robots for early speech and language assessment

   https://doi.org/10.302/1431402  

   Yeung, Gary; Afshan, Amber; Quintero, Marlen; Martin, Alejandra; Spaulding, Samuel; Park, Hae Won; Bailey, Alison; Breazeal, Cynthia; and Alwan, Abeer. (April 2019, 2019 Annual Meeting of the American Educational Research Association (AERA))

   This pilot study investigated the feasibility of implementing child-friendly robots for administering clinical and educational assessments with young children. JIBO, a social robot, was used as a new interface to administer a letter and number naming task and the 3rd Goldman Fristoe Test of Articulation (GFTA-3). The reason for using these assessment materials is to develop robust automatic speech recognition (ASR) and automated social interaction systems that can aid in administering such assessments more efficiently. The voice of JIBO simulates interaction with a peer, and images and playful transitions are displayed on JIBO’s face/screen. Several preliminary observations with 15 pre-kindergarten and 18 kindergarten students included the rate of task completion and strategies to increase student participation. Changes to the length and prompt delivery of the assessment protocol were considered based on these observations, and further observations are planned for future work with an additional cohort of 43 prekindergarten and 50 kindergarten students. Recommendations are given to inform future implementations and analyses. 

   more » « less
4. Towards Better Meta-Initialization with Task Augmentation for Kindergarten-Aged Speech Recognition

   https://doi.org/10.1109/ICASSP43922.2022.9747599  

   Zhu, Yunzheng; Fan, Ruchao; Alwan, Abeer (May 2022, Proceedings of the IEEE ICASSP)

   Children’s automatic speech recognition (ASR) is always difficult due to, in part, the data scarcity problem, especially for kindergarten-aged kids. When data are scarce, the model might overfit to the training data, and hence good starting points for training are essential. Recently, meta-learning was proposed to learn model initialization (MI) for ASR tasks of different languages. This method leads to good performance when the model is adapted to an unseen language. How-ever, MI is vulnerable to overfitting on training tasks (learner overfitting). It is also unknown whether MI generalizes to other low-resource tasks. In this paper, we validate the effectiveness of MI in children’s ASR and attempt to alleviate the problem of learner overfitting. To achieve model-agnostic meta-learning (MAML), we regard children’s speech at each age as a different task. In terms of learner overfitting, we propose a task-level augmentation method by simulating new ages using frequency warping techniques. Detailed experiments are conducted to show the impact of task augmentation on each age for kindergarten-aged speech. As a result, our approach achieves a relative word error rate (WER) improvement of 51% over the baseline system with no augmentation or initialization. 

   more » « less
5. Development of vowel acoustics and subglottal resonances in American English-speaking children: A longitudinal Study

   https://doi.org/10.1121/10.0016294  

   Shetty, Vishwas; Lulich, Steven M.; Palo, Pertti; Alwan, Abeer (October 2022, The Journal of the Acoustical Society of America)

   Acoustic analysis of typically developing elementary school-aged (prepubertal) children’s speech has been primarily performed on cross-sectional data in the past. Few studies have examined longitudinal data in this age group. For this presentation, we analyze the developmental changes in the acoustic properties of children’s speech using data collected longitudinally over four years (from first grade to fourth grade). Four male and four female children participated in this study. Data were collected once every year for each child. Using these data, we measured the four-year development of subglottal acoustics (first two subglottal resonances) and vowel acoustics (first four formants and fundamental frequency). Subglottal acoustic measurements are relatively independent of context, and average values were obtained for each child in each year. Vowel acoustics measurements were made for seven vowels (i, ɪ, ɛ, æ, ʌ, ɑ, u), each occurring in two different words in the stressed syllable. We investigated the correlations between the children’s subglottal acoustics, vowel acoustics, and growth-related variables such as standing height, sitting height, and chronological age. Gender-, vowel-, and child-specific analyses were carried out in order to shed light on how typically developing speech acoustics depend on such variables. [Work supported, in part, by the NSF.] 

   more » « less