More Like this

1. Building Language-Agnostic Grounded Language Learning Systems.

   https://doi.org/10.1109/RO-MAN46459.2019.8956449  

   Kery, Caroline; Pillai, Nisha; Matuszek, Cynthia; Ferraro, Francis (October 2019, IEEE International Conference on Robot and Human Interactive Communication)

   null (Ed.)

   Learning the meaning of grounded language---language that references a robot’s physical environment and perceptual data---is an important and increasingly widely studied problem in robotics and human-robot interaction. However, with a few exceptions, research in robotics has focused on learning groundings for a single natural language pertaining to rich perceptual data. We present experiments on taking an existing natural language grounding system designed for English and applying it to a novel multilingual corpus of descriptions of objects paired with RGB-D perceptual data. We demonstrate that this specific approach transfers well to different languages, but also present possible design constraints to consider for grounded language learning systems intended for robots that will function in a variety of linguistic settings. 

   more » « less
2. Profiling Programming Language Learning

   https://doi.org/10.1145/3649812  

   Crichton, Will; Krishnamurthi, Shriram (April 2024, Proceedings of the ACM on Programming Languages)

   This paper documents a year-long experiment to “profile” the process of learning a programming language: gathering data to understand what makes a language hard to learn, and using that data to improve the learning process. We added interactive quizzes to The Rust Programming Language, the official textbook for learning Rust. Over 13 months, 62,526 readers answered questions 1,140,202 times. First, we analyze the trajectories of readers. We find that many readers drop-out of the book early when faced with difficult language concepts like Rust’s ownership types. Second, we use classical test theory and item response theory to analyze the characteristics of quiz questions. We find that better questions are more conceptual in nature, such as asking why a program does not compile vs. whether a program compiles. Third, we performed 12 interventions into the book to help readers with difficult questions. We find that on average, interventions improved quiz scores on the targeted questions by +20 

   more » « less
3. Learning to Color from Language

   Iyyer, Mohit; Manjunatha, Varun; Boyd-Graber, Jordan; Davis, Larry (January 2018, North American Association of Computational Linguistics)
4. Learning from Natural Language Feedback

   Chen, A; Scheurer, J; Campos, JA; Korbak, T; Chan, JS; Bowman, SR; Cho, K; Perez, E (February 2024, Transactions on machine learning research)

   The potential for pre-trained large language models (LLMs) to use natural language feedback at inference time has been an exciting recent development. We build upon this observation by formalizing an algorithm for learning from natural language feedback at training time instead, which we call Imitation learning from Language Feedback (ILF). ILF requires only a small amount of human-written feedback during training and does not require the same feedback at test time, making it both user-friendly and sample-efficient. We further show that ILF can be seen as a form of minimizing the KL divergence to the target distribution and demonstrate proof-of-concepts on text summarization and program synthesis tasks. For code generation, ILF improves a Codegen-Mono 6.1B model’s pass@1 rate from 22% to 36% on the MBPP benchmark, outperforming both fine-tuning on MBPP and on human- written repaired programs. For summarization, we show that ILF can be combined with learning from human preferences to improve a GPT-3 model’s summarization performance to be comparable to human quality, outperforming fine-tuning on human-written summaries. Overall, our results suggest that ILF is both more effective and sample-efficient than training exclusively on demonstrations for improving an LLM’s performance on a variety of tasks. 

   more » « less
5. Learning Planning Abstractions from Language

   Liu, Weiyu; Chen, Geng; Hsu, Joy; Mao, Jiayuan; Wu, Jiajun (May 2024, International Conference on Learning Representations (ICLR))