More Like this

1. Rethinking bilingual enhancement effects in associative learning of foreign language vocabulary: The role of proficiency in the mediating language

   https://doi.org/10.1016/j.jml.2020.104155  

   Tsuboi, Naoko ; Francis, Wendy S. ( December 2020 , Journal of Memory and Language)
2. A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning

   Kébé, Gaoussou Youssouf ; Higgins, Padraig ; Jenkins, Patrick ; Darvish, Kasra ; Sachdeva, Rishabh ; Barron, Ryan ; Winder, John ; Engel, Don ; Raff, Edward ; Ferraro, Francis ; et al ( December 2021 , Advances in neural information processing systems)

   Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results. 

   more » « less
3. Capturing the variation in language experience to understand language processing and learning

   https://doi.org/10.1075/lia.20018.kro  

   Kroll, Judith F. ; Takahesu Tabori, Andrea ; Navarro-Torres, Christian ( July 2021 , Language, Interaction and Acquisition)

   Abstract A goal of early research on language processing was to characterize what is universal about language. Much of the past research focused on native speakers because the native language has been considered as providing privileged truths about acquisition, comprehension, and production. Populations or circumstances that deviated from these idealized norms were of interest but not regarded as essential to our understanding of language. In the past two decades, there has been a marked change in our understanding of how variation in language experience may inform the central and enduring questions about language. There is now evidence for significant plasticity in language learning beyond early childhood, and variation in language experience has been shown to influence both language learning and processing. In this paper, we feature what we take to be the most exciting recent new discoveries suggesting that variation in language experience provides a lens into the linguistic, cognitive, and neural mechanisms that enable language processing. 

   more » « less
4. Building Language-Agnostic Grounded Language Learning Systems.

   https://doi.org/10.1109/RO-MAN46459.2019.8956449  

   Kery, Caroline ; Pillai, Nisha ; Matuszek, Cynthia ; Ferraro, Francis ( October 2019 , IEEE International Conference on Robot and Human Interactive Communication)

   null (Ed.)

   Learning the meaning of grounded language---language that references a robot’s physical environment and perceptual data---is an important and increasingly widely studied problem in robotics and human-robot interaction. However, with a few exceptions, research in robotics has focused on learning groundings for a single natural language pertaining to rich perceptual data. We present experiments on taking an existing natural language grounding system designed for English and applying it to a novel multilingual corpus of descriptions of objects paired with RGB-D perceptual data. We demonstrate that this specific approach transfers well to different languages, but also present possible design constraints to consider for grounded language learning systems intended for robots that will function in a variety of linguistic settings. 

   more » « less
5. A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning

   Kebe, Gaoussou Y. ; Higgins, Padraig ; Jenkins, Patrick ; Darvish, Kasra ; Sachdeva, Rishabh ; Barron, Ryan ; Winder, John ; Engel, Don ; Raff, Edward ; Ferraro, Francis ; et al ( December 2021 , Advances in neural information processing systems)

   Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results. 

   more » « less