skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Towards Assistive Robotic Pick and Place in Open World Environments
Assistive robot manipulators must be able to autonomously pick and place a wide range of novel objects to be truly useful. However, current assistive robots lack this capability. Additionally, assistive systems need to have an interface that is easy to learn, to use, and to understand. This paper takes a step forward in this direction. We present a robot system comprised of a robotic arm and a mobility scooter that provides both pick-and-drop and pick-and-place functionality for open world environments without modeling the objects or environment. The system uses a laser pointer to directly select an object in the world, with feedback to the user via projecting an interface into the world. Our evaluation over several experimental scenarios shows a significant improvement in both runtime and grasp success rate relative to a baseline from the literature, and furthermore demonstrates accurate pick and place capabilities for tabletop scenarios.  more » « less
Award ID(s):
1763469
PAR ID:
10179635
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
Proceedings of the International Symposium on Robotics Research (ISRR)
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; (, Springer Nature Singapore)
    In social robotics, a pivotal focus is enabling robots to engage with humans in a more natural and seamless manner. The emergence of advanced large language models (LLMs) has driven significant advancements in integrating natural language understanding capabilities into social robots. This paper presents a system for speech-guided sequential planning in pick and place tasks, which are found across a range of application areas. The proposed system uses Large Language Model Meta AI (Llama3) to interpret voice commands by extracting essential details through parsing and decoding the commands into sequential actions. These actions are sent to DRL-VO, a learning-based control policy built on the Robot Operating System (ROS) that allows a robot to autonomously navigate through social spaces with static infrastructure and crowds of people. We demonstrate the effectiveness of the system in simulation experiment using Turtlebot 2 in ROS1 and Turtlebot 3 in ROS2. We conduct hardware trials using a Clearpath Robotics Jackal UGV, highlighting its potential for real-world deployment in scenarios requiring flexible and interactive robotic behaviors. 
    more » « less
  2. ; ; ; ; ; ; ; ; (, Proceedings of the International Conference on Robotics and Automation (ICRA))
    Natural language understanding for robotics can require substantial domain- and platform-specific engineering. For example, for mobile robots to pick-and-place objects in an environment to satisfy human commands, we can specify the language humans use to issue such commands, and connect concept words like red can to physical object properties. One way to alleviate this engineering for a new domain is to enable robots in human environments to adapt dynamically -- continually learning new language constructions and perceptual concepts. In this work, we present an end-to-end pipeline for translating natural language commands to discrete robot actions, and use clarification dialogs to jointly improve language parsing and concept grounding. We train and evaluate this agent in a virtual setting on Amazon Mechanical Turk, and we transfer the learned agent to a physical robot platform to demonstrate it in the real world. 
    more » « less
  3. ; ; ; (, IEEE)
    This work focuses on the problem of robotic picking in challenging multi-object scenarios. These scenarios include difficult-to-pick objects (e.g., too small, too flat objects) and challenging conditions (e.g., objects obstructed by other objects and/or the environment). To solve these challenges, we leverage four dexterous picking skills inspired by human manipulation techniques and propose methods based on deep neural networks that predict when and how to apply the skills based on the shape of the objects, their relative locations to each other, and the environmental factors. We utilize a compliant, under-actuated hand to reliably apply the identified skills in an open-loop manner. The capabilities of the proposed system are evaluated through a series of real-world experiments, comprising 45 trials with 150+ grasps, to assess its reliability and robustness, particularly in cluttered settings. The videos of all experiments are provided at https://dexterouspicking.wpi.edu/. This research helps bridge the gap between human and robotic grasping, showcasing promising results in various practical scenarios. 
    more » « less
  4. ; ; ; ; ; (, Proceedings of the AAAI Conference on Artificial Intelligence)
    Collaborative robotics requires effective communication between a robot and a human partner. This work proposes a set of interpretive principles for how a robotic arm can use pointing actions to communicate task information to people by extending existing models from the related literature. These principles are evaluated through studies where English-speaking human subjects view animations of simulated robots instructing pick-and-place tasks. The evaluation distinguishes two classes of pointing actions that arise in pick-and-place tasks: referential pointing (identifying objects) and locating pointing (identifying locations). The study indicates that human subjects show greater flexibility in interpreting the intent of referential pointing compared to locating pointing, which needs to be more deliberate. The results also demonstrate the effects of variation in the environment and task context on the interpretation of pointing. Our corpus, experiments and design principles advance models of context, common sense reasoning and communication in embodied communication. 
    more » « less
  5. ; ; ; ; (, The International Journal of Robotics Research)
    Robotic pick and place tasks are symmetric under translations and rotations of both the object to be picked and the desired place pose. For example, if the pick object is rotated or translated, then the optimal pick action should also rotate or translate. The same is true for the place pose; if the desired place pose changes, then the place action should also transform accordingly. A recently proposed pick and place framework known as Transporter Net (Zeng, Florence, Tompson, Welker, Chien, Attarian, Armstrong, Krasin, Duong, Sindhwani et al., 2021) captures some of these symmetries, but not all. This paper analytically studies the symmetries present in planar robotic pick and place and proposes a method of incorporating equivariant neural models into Transporter Net in a way that captures all symmetries. The new model, which we call Equivariant Transporter Net, is equivariant to both pick and place symmetries and can immediately generalize pick and place knowledge to different pick and place poses. We evaluate the new model empirically and show that it is much more sample-efficient than the non-symmetric version, resulting in a system that can imitate demonstrated pick and place behavior using very few human demonstrations on a variety of imitation learning tasks. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email