More Like this

1. Transferring Implicit Knowledge of Non-Visual Object Properties Across Heterogeneous Robot Morphologies

   https://doi.org/10.1109/ICRA48891.2023.10160811  

   Tatiya G.; Francis J.; and Sinapov, J. (May 2023, IEEE International Conference on Robotics and Automation)

   Abstract— Humans leverage multiple sensor modalities when interacting with objects and discovering their intrinsic properties. Using the visual modality alone is insufficient for deriving intuition behind object properties (e.g., which of two boxes is heavier), making it essential to consider non-visual modalities as well, such as the tactile and auditory. Whereas robots may leverage various modalities to obtain object property understanding via learned exploratory interactions with objects (e.g., grasping, lifting, and shaking behaviors), challenges remain: the implicit knowledge acquired by one robot via object exploration cannot be directly leveraged by another robot with different morphology, because the sensor models, observed data distributions, and interaction capabilities are different across these different robot configurations. To avoid the costly process of learning interactive object perception tasks from scratch, we propose a multi-stage projection framework for each new robot for transferring implicit knowledge of object properties across heterogeneous robot morphologies. We evaluate our approach on the object-property recognition and object-identity recognition tasks, using a dataset containing two heterogeneous robots that perform 7,600 object interactions. Results indicate that knowledge can be transferred across robots, such that a newly-deployed robot can bootstrap its recognition models without exhaustively exploring all objects. We also propose a data augmentation technique and show that this technique improves the generalization of models. We release code, datasets, and additional results, here: https: //github.com/gtatiya/Implicit-Knowledge-Transfer. 

   more » « less
2. Visual Inverse Kinematics: Finding Feasible Robot Poses under Kinematic and Vision Constraints

   Wu, Liangting; Tron, Roberto (July 2025, W Telekomunikacji)

   We introduce Visual Inverse Kinematics (VIK), which finds kinematically feasible joint configurations that satisfy vision-based constraints, bridging the gap between inverse kinematics (IK) and visual servoing (VS). Unlike IK, no explicit end-effector pose is given, and unlike VS, exact image measurements may not be available. In this work, we develop a formulation of the VIK problem with a field of view (FoV) constraint, enforcing the visibility of an object from a camera on the robot. Our proposed solution introduces a virtual kinematic chain that connects the physical robot and the object, transforming the FoV constraint into a joint angle kinematic constraint. Along the way, we introduce multiple vision-based cost functions to fulfill different objectives. We solve this formulation of the VIK problem using a method that involves a semidefinite program (SDP) constraint followed by a rank minimization algorithm. The performance of this method for solving the VIK problem is validated through simulations. 

   more » « less
3. Design and Analysis of a Wearable Robotic Forearm

   https://doi.org/10.1109/ICRA.2018.8461212  

   Vatsal, Vighnesh; Hoffman, Guy (May 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA))

   This paper presents the design of a wearable robotic forearm for close-range human-robot collaboration. The robot's function is to serve as a lightweight supernumerary third arm for shared workspace activities. We present a functional prototype resulting from an iterative design process including several user studies. An analysis of the robot's kinematics shows an increase in reachable workspace by 246 % compared to the natural human reach. The robot's degrees of freedom and range of motion support a variety of usage scenarios with the robot as a collaborative tool, including self-handovers, fetching objects while the human's hands are occupied, assisting human-human collaboration, and stabilizing an object. We analyze the bio-mechanical loads for these scenarios and find that the design is able to operate within human ergonomic wear limits. We then report on a pilot human-robot interaction study that indicates robot autonomy is more task-time efficient and preferred by users when compared to direct voice-control. These results suggest that the design presented here is a promising configuration for a lightweight wearable robotic augmentation device, and can serve as a basis for further research into human-wearable collaboration. 

   more » « less
4. Learning Rope Manipulation Policies Using Dense Object Descriptors Trained on Synthetic Depth Data

   https://doi.org/10.1109/ICRA40945.2020.9197121  

   Sundaresan, Priya; Grannen, Jennifer; Thananjeyan, Brijen; Balakrishna, Ashwin; Laskey, Michael; Stone, Kevin; Gonzalez, Joseph E.; Goldberg, Ken (May 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA))

   null (Ed.)

   Robotic manipulation of deformable 1D objects such as ropes, cables, and hoses is challenging due to the lack of high-fidelity analytic models and large configuration spaces. Furthermore, learning end-to-end manipulation policies directly from images and physical interaction requires significant time on a robot and can fail to generalize across tasks. We address these challenges using interpretable deep visual representations for rope, extending recent work on dense object descriptors for robot manipulation. This facilitates the design of interpretable and transferable geometric policies built on top of the learned representations, decoupling visual reasoning and control. We present an approach that learns point-pair correspondences between initial and goal rope configurations, which implicitly encodes geometric structure, entirely in simulation from synthetic depth images. We demonstrate that the learned representation - dense depth object descriptors (DDODs) - can be used to manipulate a real rope into a variety of different arrangements either by learning from demonstrations or using interpretable geometric policies. In 50 trials of a knot-tying task with the ABB YuMi Robot, the system achieves a 66% knot-tying success rate from previously unseen configurations. See https://tinyurl.com/rope-learning for supplementary material and videos. 

   more » « less
5. Finding fallen objects via asynchronous audio-visual integration

   https://doi.org/10.1109/CVPR52688.2022.01027  

   Gan, C.; Gu, Y.; Zhou, S.; Schwartz, J.; Alter, S.; Traer, J.; Gutfreund, D.; Tenenbaum, J.B.; McDermott, J.H.; Torralba, A. (July 2022, IEEE Conference on Computer Vision and Pattern Recognition)

   The way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be integrated, as when we hear an object dropped on the floor and then must find it. In this paper, we introduce a setting in which to study multi-modal object localization in 3D virtual environments. An object is dropped somewhere in a room. An embodied robot agent, equipped with a camera and microphone, must determine what object has been dropped -- and where -- by combining audio and visual signals with knowledge of the underlying physics. To study this problem, we have generated a large-scale dataset -- the Fallen Objects dataset -- that includes 8000 instances of 30 physical object categories in 64 rooms. The dataset uses the ThreeDWorld Platform that can simulate physics-based impact sounds and complex physical interactions between objects in a photorealistic setting. As a first step toward addressing this challenge, we develop a set of embodied agent baselines, based on imitation learning, reinforcement learning, and modular planning, and perform an in-depth analysis of the challenge of this new task. 

   more » « less