Search for: All records

Abstract Robotic task execution is susceptible to errors due to uncertainties in environmental conditions and physical systems, often resulting in undesirable or potentially hazardous outcomes. Enhancing robot performance necessitates the ability to enable robots to comprehend and rectify abnormal executions. However, contemporary robots possess limited cognitive capabilities for environment comprehension and task reasoning, rendering the self-diagnosis and correction of anomalies a formidable challenge. To address this issue, we present a novel approach known as “human-to-robot attention transfer” (H2R-AT) designed to facilitate human assistance in diagnosing robot abnormalities. H2R-AT leverages a novel stacked neural networks model and an attention mechanism to transfer attention from humans to robots, aiding in the diagnosis of abnormalities. In this process, humans verbally express concerns regarding abnormal robot executions, enabling the robot to discern the nature and location of these concerns, thereby facilitating corrective actions. To evaluate the efficacy of H2R-AT, we devised two representative task scenarios: “serving water for a human in a kitchen” and “picking up a defective gear in a factory,” both involving abnormal behaviors, within the simulation platform craihri. We recruited 252 volunteers who provided approximately 12,000 verbal reminders for learning and testing the attention transfer model H2R-AT. Achieving an accuracy of 73.68% in attention transfer and 90.75% in correctly implementing suggested corrections, the model demonstrated an overall accuracy of 66.86% in preventing robot execution failures. These results affirm the effectiveness of H2R-AT in averting robot failures during task execution. 

Deep Reinforcement Learning (DRL) has shown its capability to solve the high degrees of freedom in control and the complex interaction with the object in the multi-finger dexterous in-hand manipulation tasks. Current DRL approaches lack behavior constraints during the learning process, leading to aggressive and unstable policies that are insufficient for safety-critical in-hand manipulation tasks. The centralized learning strategy also limits the flexibility to fine-tune each robot finger's behavior. This work proposes the Finger-specific Multi-agent Shadow Critic Consensus (FMSC) method, which models the in-hand manipulation as a multi-agent collaboration task where each finger is an individual agent and trains the policies for the fingers to achieve a consensus across the critic networks through the Information Sharing (IS) across the neighboring agents and finger-specific stable manipulation objectives based on the state-action occupancy measure, a general utility of DRL that is approximated during the learning process. The methods are evaluated in two in-hand manipulation tasks on the Shadow Hand. The results show that FMSC+IS converges faster in training, achieving a comparable success rate and much better manipulation stability than conventional DRL methods.