Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results.
more »
« less
A Spoken Language Dataset of Descriptions for Speech-Based Grounded Language Learning
Grounded language acquisition is a major area of research combining aspects of natural language processing, computer vision, and signal processing, compounded by domain issues requiring sample efficiency and other deployment constraints. In this work, we present a multimodal dataset of RGB+depth objects with spoken as well as textual descriptions. We analyze the differences between the two types of descriptive language and our experiments demonstrate that the different modalities affect learning. This will enable researchers studying the intersection of robotics, NLP, and HCI to better investigate how the multiple modalities of image, depth, text, speech, and transcription interact, as well as how differences in the vernacular of these modalities impact results.
more »
« less
- NSF-PAR ID:
- 10296649
- Date Published:
- Journal Name:
- Advances in neural information processing systems
- ISSN:
- 1049-5258
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Abstract A goal of early research on language processing was to characterize what is universal about language. Much of the past research focused on native speakers because the native language has been considered as providing privileged truths about acquisition, comprehension, and production. Populations or circumstances that deviated from these idealized norms were of interest but not regarded as essential to our understanding of language. In the past two decades, there has been a marked change in our understanding of how variation in language experience may inform the central and enduring questions about language. There is now evidence for significant plasticity in language learning beyond early childhood, and variation in language experience has been shown to influence both language learning and processing. In this paper, we feature what we take to be the most exciting recent new discoveries suggesting that variation in language experience provides a lens into the linguistic, cognitive, and neural mechanisms that enable language processing.more » « less
-
Kail, Michèle ; null (Ed.)A goal of early research on language processing was to characterize what is universal about language. Much of the past research focused on native speakers because the native language has been considered as providing privileged truths about acquisition, comprehension, and production. Populations or circumstances that deviated from these idealized norms were of interest but not regarded as essential to our understanding of language. In the past two decades, there has been a marked change in our understanding of how variation in language experience may inform the central and enduring questions about language. There is now evidence for significant plasticity in language learning beyond early childhood, and variation in language experience has been shown to influence both language learning and processing. In this paper, we feature what we take to be the most exciting recent new discoveries suggesting that variation in language experience provides a lens into the linguistic, cognitive, and neural mechanisms that enable language processing.more » « less
-
more » « less
Abstract The study of how bilingualism is linked to cognitive processing, including executive functioning, has historically focused on comparing bilinguals to monolinguals across a range of tasks. These group comparisons presume to capture relatively stable cognitive traits and have revealed important insights about the architecture of the language processing system that could not have been gleaned from studying monolinguals alone. However, there are drawbacks to using a group-comparison, or Traits, approach. In this theoretical review, we outline some limitations of treating executive functions as stable traits and of treating bilinguals as a uniform group when compared to monolinguals. To build on what we have learned from group comparisons, we advocate for an emerging complementary approach to the question of cognition and bilingualism. Using an approach that compares bilinguals to themselves under different linguistic or cognitive contexts allows researchers to ask questions about how language and cognitive processes interact based on dynamically fluctuating cognitive and neural states. A States approach, which has already been used by bilingualism researchers, allows for cause-and-effect hypotheses and shifts our focus from questions of group differences to questions of how varied linguistic environments influence cognitive operations in the moment and how fluctuations in cognitive engagement impact language processing.
-
more » « less
Abstract Language comprehension involves integrating low-level sensory inputs into a hierarchy of increasingly high-level features. Prior work studied brain representations of different levels of the language hierarchy, but has not determined whether these brain representations are shared between written and spoken language. To address this issue, we analyze fMRI BOLD data that were recorded while participants read and listened to the same narratives in each modality. Levels of the language hierarchy are operationalized as timescales, where each timescale refers to a set of spectral components of a language stimulus. Voxelwise encoding models are used to determine where different timescales are represented across the cerebral cortex, for each modality separately. These models reveal that between the two modalities timescale representations are organized similarly across the cortical surface. Our results suggest that, after low-level sensory processing, language integration proceeds similarly regardless of stimulus modality.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
