More Like this

1. Learning of Complex-Structured Tasks from Verbal Instruction

   https://doi.org/10.1109/Humanoids43949.2019.9035032  

   Nicolescu, M; Arnold, N; Blankenburg, J; Feil-Seifer, D; Balajee Banisetty, S; Nicolescu, M; Palmer, A; Monteverde, T (October 2020, IEEERAS International Conference on Humanoid Robots)

   null (Ed.)

   This paper presents a novel approach to robot task learning from language-based instructions, which focuses on increasing the complexity of task representations that can be taught through verbal instruction. The major proposed contribution is the development of a framework for directly mapping a complex verbal instruction to an executable task representation, from a single training experience. The method can handle the following types of complexities: 1) instructions that use conjunctions to convey complex execution constraints (such as alternative paths of execution, sequential or nonordering constraints, as well as hierarchical representations) and 2) instructions that use prepositions and multiple adjectives to specify action/object parameters relevant for the task. Specific algorithms have been developed for handling conjunctions, adjectives and prepositions as well as for translating the parsed instructions into parameterized executable task representations. The paper describes validation experiments with a PR2 humanoid robot learning new tasks from verbal instruction, as well as an additional range of utterances that can be parsed into executable controllers by the proposed system. 

   more » « less
2. Using Natural Language for Reward Shaping in Reinforcement Learning

   Goyal, P.; Niekum, S.; Mooney, R.J. (August 2019, Proceedings of the 28th International Joint Conference on Artificial Intelligence)

   Recent reinforcement learning (RL) approaches have shown strong performance in complex domains such as Atari games, but are often highly sample inefficient. A common approach to reduce interaction time with the environment is to use reward shaping, which involves carefully designing reward functions that provide the agent intermediate rewards for progress towards the goal. However, designing appropriate shaping rewards is known to be difficult as well as time-consuming. In this work, we address this problem by using natural language instructions to perform reward shaping. We propose the LanguagE-Action Reward Network (LEARN), a framework that maps free-form natural language instructions to intermediate rewards based on actions taken by the agent. These intermediate language-based rewards can seamlessly be integrated into any standard reinforcement learning algorithm. We experiment with Montezuma’s Revenge from the Atari Learning Environment, a popular benchmark in RL. Our experiments on a diverse set of 15 tasks demonstrate that, for the same number of interactions with the environment, language-based rewards lead to successful completion of the task 60 % more often on average, compared to learning without language. 

   more » « less
3. Benefits of Testing and Production for Learning Turkish As a New Language

   https://doi.org/10.1111/lang.12602  

   Rose, Maya C.; Brooks, Patricia J.; Lodhi, Arshia K.; Cortez, Angela (July 2023, Language Learning)

   Abstract This study examined putative benefits of testing and production for learning new languages. Undergraduates (N= 156) were exposed to Turkish spoken dialogues under varying learning conditions (retrieval practice, comprehension, verbal repetition) in a computer‐assisted language learning session. Participants completed pre‐ and posttests of number‐ and case‐marking comprehension, a vocabulary test, and an explicit awareness questionnaire. Controlling for nonverbal ability and pretest scores, the retrieval‐practice group performed highest overall. For number/case marking, the comprehension and retrieval‐practice groups outperformed the verbal‐repetition group, suggesting benefits of either recognition‐ or recall‐based testing. For vocabulary, the verbal‐repetition and retrieval‐practice groups outperformed the comprehension group, indicating benefits of overt production. Case marking was easier to learn than number marking, suggesting advantages for learning word‐final inflections. Explicit awareness correlated with comprehension accuracy, yet some participants demonstrated above‐chance comprehension without showing awareness. Findings indicate the value of incorporating both practice tests and overt production in language pedagogy. 

   more » « less
4. Using a picture (or a thousand words) for supporting spatial knowledge of a complex virtual environment

   https://doi.org/10.1186/s41235-023-00503-z  

   Jaeger, Allison J.; Weisberg, Steven M.; Nazareth, Alina; Newcombe, Nora S. (July 2023, Cognitive Research: Principles and Implications)

   Abstract External representations powerfully support and augment complex human behavior. When navigating, people often consult external representations to help them find the way to go, but do maps or verbal instructions improve spatial knowledge or support effective wayfinding? Here, we examine spatial knowledge with and without external representations in two studies where participants learn a complex virtual environment. In the first study, we asked participants to generate their own maps or verbal instructions, partway through learning. We found no evidence of improved spatial knowledge in a pointing task requiring participants to infer the direction between two targets, either on the same route or on different routes, and no differences between groups in accurately recreating a map of the target landmarks. However, as a methodological note, pointing was correlated with the accuracy of the maps that participants drew. In the second study, participants had access to an accurate map or set of verbal instructions that they could study while learning the layout of target landmarks. Again, we found no evidence of differentially improved spatial knowledge in the pointing task, although we did find that the map group could recreate a map of the target landmarks more accurately. However, overall improvement was high. There was evidence that the nature of improvement across all conditions was specific to initial navigation ability levels. Our findings add to a mixed literature on the role of external representations for navigation and suggest that more substantial intervention—more scaffolding, explicit training, enhanced visualization, perhaps with personalized sequencing—may be necessary to improve navigation ability. 

   more » « less
5. What categorical induction variability reveals about typical and atypical development

   https://doi.org/10.1017/S0305000920000392  

   TECOULESCO, Lisa; FEIN, Deborah; NAIGLES, Letitia R. (May 2021, Journal of Child Language)

   null (Ed.)

   Abstract Categorical induction abilities are robust in typically developing (TD) preschoolers, while children with Autism Spectrum Disorders (ASD) frequently perform inconsistently on tasks asking for the transference of traits from a known category member to a new example based on shared category membership. Here, TD five-year-olds and six-year-olds with ASD participated in a categorical induction task; the TD children performed significantly better and more consistently than the children with ASD. Concurrent verbal and nonverbal tests were not significant correlates; however, the TD children's shape bias performance at two years of age was significantly positively predictive of categorical induction performance at age five. The shape bias, the tendency to extend a novel label to other objects of the same shape during word learning, appears linked with categorical induction ability in TD children, suggesting a common underlying skill and consistent developmental trajectory. Word learning and categorical induction appear uncoupled in children with ASD. 

   more » « less