More Like this

1. Learning Object Attributes with Category-Free Grounded Language from Deep Featurization

   Richards, Luke E.; Darvish, Kasra; Matuszek, Cynthia (September 2020, Proceedings of the IEEERSJ International Conference on Intelligent Robots and Systems)

   null (Ed.)

   While grounded language learning, or learning the meaning of language with respect to the physical world in which a robot operates, is a major area in human-robot interaction studies, most research occurs in closed worlds or domain-constrained settings. We present a system in which language is grounded in visual percepts without using categorical constraints by combining CNN-based visual featurization with natural language labels. We demonstrate results comparable to those achieved using handcrafted features for specific traits, a step towards moving language grounding into the space of fully open world recognition. 

   more » « less
2. Building Language-Agnostic Grounded Language Learning Systems.

   https://doi.org/10.1109/RO-MAN46459.2019.8956449  

   Kery, Caroline; Pillai, Nisha; Matuszek, Cynthia; Ferraro, Francis (October 2019, IEEE International Conference on Robot and Human Interactive Communication)

   null (Ed.)

   Learning the meaning of grounded language---language that references a robot’s physical environment and perceptual data---is an important and increasingly widely studied problem in robotics and human-robot interaction. However, with a few exceptions, research in robotics has focused on learning groundings for a single natural language pertaining to rich perceptual data. We present experiments on taking an existing natural language grounding system designed for English and applying it to a novel multilingual corpus of descriptions of objects paired with RGB-D perceptual data. We demonstrate that this specific approach transfers well to different languages, but also present possible design constraints to consider for grounded language learning systems intended for robots that will function in a variety of linguistic settings. 

   more » « less
3. Robots That Use Language

   https://doi.org/10.1146/annurev-control-101119-071628  

   Tellex, Stefanie; Gopalan, Nakul; Kress-Gazit, Hadas; Matuszek, Cynthia (May 2020, Annual Review of Control, Robotics, and Autonomous Systems)

   This article surveys the use of natural language in robotics from a robotics point of view. To use human language, robots must map words to aspects of the physical world, mediated by the robot's sensors and actuators. This problem differs from other natural language processing domains due to the need to ground the language to noisy percepts and physical actions. Here, we describe central aspects of language use by robots, including understanding natural language requests, using language to drive learning about the physical world, and engaging in collaborative dialogue with a human partner. We describe common approaches, roughly divided into learning methods, logic-based methods, and methods that focus on questions of human–robot interaction. Finally, we describe several application domains for language-using robots. 

   more » « less
4. Comparing Strategies for Robot Communication of Role-Grounded Moral Norms

   https://doi.org/10.1145/3434074.3447185  

   Wen, Ruchen; Kim, Boyoung; Phillips, Elizabeth; Zhu, Qin; Williams, Tom (March 2021, ACM/IEEE International Conference on Human-Robot Interaction)

   null (Ed.)

   Because robots are perceived as moral agents, they hold significant persuasive power over humans. It is thus crucial for robots to behave in accordance with human systems of morality and to use effective strategies for human-robot moral communication. In this work, we evaluate two moral communication strategies: a norm-based strategy grounded in deontological ethics, and a role-based strategy grounded in role ethics, in order to test the effectiveness of these two strategies in encouraging compliance with norms grounded in role expectations. Our results suggest two major findings: (1) reflective exercises may increase the efficacy of role-based moral language and (2) opportunities for moral practice following robots’ use of moral language may facilitate role-centered moral cultivation. 

   more » « less
5. Inferring Compact Representations for Efficient Natural Language Understanding of Robot Instructions

   https://doi.org/10.1109/ICRA.2019.8793667  

   Patki, Siddharth; Daniele, Andrea F.; Walter, Matthew R.; Howard, Thomas M. (May 2019, Proceedings of the International Conference on Robotics and Automation (ICRA))

   The speed and accuracy with which robots are able to interpret natural language is fundamental to realizing effective human-robot interaction. A great deal of attention has been paid to developing models and approximate inference algorithms that improve the efficiency of language understanding. However, existing methods still attempt to reason over a representation of the environment that is flat and unnecessarily detailed, which limits scalability. An open problem is then to develop methods capable of producing the most compact environment model sufficient for accurate and efficient natural language understanding. We propose a model that leverages environment-related information encoded within instructions to identify the subset of observations and perceptual classifiers necessary to perceive a succinct, instruction-specific environment representation. The framework uses three probabilistic graphical models trained from a corpus of annotated instructions to infer salient scene semantics, perceptual classifiers, and grounded symbols. Experimental results on two robots operating in different environments demonstrate that by exploiting the content and the structure of the instructions, our method learns compact environment representations that significantly improve the efficiency of natural language symbol grounding. 

   more » « less