skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: iOS Based Pose Estimation and Gesture Recognition for Robot Manipulation
Interfacing between robots and humans has come a long way in the past few years, and new methods for smart, robust interaction are needed. Typically, a technician has to program a routine for a robot in order for the robot to be useful. This puts up a significant barrier to entry into the field of automating tasks using robots—not only is a technician and a computer required, but the robot is not adaptive to the immediate needs of the user. The robot is only capable of executing a pre-determined task and for any change to be made the entire system needs to be paused. This project seeks to bridge the gap between user and robot interface, creating an easy-to-use system that allows for adaptive robot control. Using a combination of computer vision and a monocular camera system and integrated LiDAR sensor on an iPhone, gesture recognition and pose estimation was conducted within an independent system to control the Baxter humanoid robot. The gathered data was sent wirelessly to the robot to be interpreted and then replay actions performed by the user.  more » « less
Award ID(s):
2125362
PAR ID:
10343335
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
2022 17th Annual System of Systems Engineering Conference (SOSE)
Page Range / eLocation ID:
438 to 443
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Ronchieri, Elisabetta; Carbone, John Carbone; Then, Patrick; Juric, Radmila (Ed.)
    Assistive robots are robots that are designed to help people with disa-bilities in their daily lives. These robots often are either too simple, and thus can-not meet the needs of the user, or are too complex, and thus become overly ex-pensive or unreliable. To mitigate this problem, we propose a system of multi-robot coordination that allows several heterogeneous robots to work coopera-tively to assist an individual. We design care task workflows that cater to the needs of a specific patient, outlining the tasks needed for the user as well as the robots that can complete each task. We then build a ROS simulation to test our coordination methods on two example workflows for PTSD care routines. Our experiments show that our coordination successfully navigates each robot to its task location at the proper time, thus showing the feasibility of these methods for multi-assistive-robot coordination. 
    more » « less
  2. Long-term deployment of a fleet of mobile robots requires reliable and secure two-way communication channels between individual robots and remote human operators for supervision and tasking. Existing open-source solutions to this problem degrade in performance in challenging real-world situations such as intermittent and low-bandwidth connectivity, do not provide security control options, and can be computationally expensive on hardware-constrained mobile robot platforms. In this paper, we present Robofleet, a lightweight open-source system which provides inter-robot communication, remote monitoring, and remote tasking for a heterogenous fleet of ROS-enabled service-mobile robots that is designed with the practical goals of resilience to network variance and security control in mind.Robofleet supports multi-user, multi-robot communication via a central server. This architecture deduplicates network traffic between robots, significantly reducing overall network load when compared with native ROS communication. This server also functions as a single entrypoint into the system, enabling security control and user authentication. Individual robots run the lightweight Robofleet client, which is responsible for exchanging messages with the Robofleet server. It automatically adapts to adverse network conditions through backpressure monitoring as well as topic-level priority control, ensuring that safety-critical messages are successfully transmitted. Finally, the system includes a web-based visualization tool that can be run on any internet-connected, browser-enabled device to monitor and control the fleet.We compare Robofleet to existing methods of robotic communication, and demonstrate that it provides superior resilience to network variance while maintaining performance that exceeds that of widely-used systems. 
    more » « less
  3. null (Ed.)
    There is a significant amount of synergy between virtual reality (VR) and the field of robotics. However, it has only been in approximately the past five years that commercial immersive VR devices have been available to developers. This new availability has led to a rapid increase in research using VR devices in the field of robotics, especially in the development of VR interfaces for operating robots. In this paper, we present a systematic review on VR interfaces for robot operation that utilize commercially available immersive VR devices. A total of 41 papers published between 2016–2020 were collected for review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Papers are discussed and categorized into five categories: (1) Visualization, which focuses on displaying data or information to operators; (2) Robot Control and Planning, which focuses on connecting human input or movement to robot movement; (3) Interaction, which focuses on the development of new interaction techniques and/or identifying best interaction practices; (4) Usability, which focuses on user experiences of VR interfaces; and (5) Infrastructure, which focuses on system architectures or software to support connecting VR and robots for interface development. Additionally, we provide future directions to continue development in VR interfaces for operating robots. 
    more » « less
  4. Soft robots employ flexible and compliant materials to perform adaptive tasks and navigate uncertain environments. However, soft robots are often unable to achieve forces and precision on the order of rigid-bodied robots. In this paper, we propose a new class of mobile soft robots that can reversibly transition between compliant and stiff states without reconfiguration. The robot can passively conform or actively control its shape, stiffen in its current configuration to function as a rigid-bodied robot, then return to its flexible form. The robotic structure consists of passive granular material surrounded by an active membrane. The membrane is composed of interconnected robotic sub-units that can control the packing density of the granular material and exploit jamming behaviors by varying the length of the interconnecting cables. Each robotic sub-unit uses a differential drive system to achieve locomotion and self-reconfigurability. We present the robot design and perform a set of locomotion and object manipulation experiments to characterize the robot's performance in soft and rigid states. We also introduce a simulation framework in which we model the jamming soft robot design and study the scalability of this class of robots. The proposed concept demonstrates the properties of both soft and rigid robots, and has the potential to bridge the gap between the two 
    more » « less
  5. Agile-legged robots have proven to be highly effective in navigating and performing tasks in complex and challenging environments, including disaster zones and industrial settings. However, these applications commonly require the capability of carrying heavy loads while maintaining dynamic motion. Therefore, this article presents a novel methodology for incorporating adaptive control into a force-based control system. Recent advancements in the control of quadruped robots show that force control can effectively realize dynamic locomotion over rough terrain. By integrating adaptive control into the force-based controller, our proposed approach can maintain the advantages of the baseline framework while adapting to significant model uncertainties and unknown terrain impact models. Experimental validation was successfully conducted on the Unitree A1 robot. With our approach, the robot can carry heavy loads (up to 50% of its weight) while performing dynamic gaits such as fast trotting and bounding across uneven terrains. 
    more » « less