More Like this

1. HO-Cap: A Capture System and Dataset for 3D Reconstruction and Pose Tracking of Hand-Object Interaction

   Wang, Jikai; Zhang, Qifan; Chao, Yu-Wei; Wen, Bowen; Guo, Xiaohu; Xiang, Yu (December 2025, NeurIPS 2025 Track on Datasets and Benchmarks (NeurIPS), 2025)

   We introduce a data capture system and a new dataset, HO-Cap, for 3D reconstruction and pose tracking of hands and objects in videos. The system leverages multiple RGB-D cameras and a HoloLens headset for data collection, avoiding the use of expensive 3D scanners or motion capture systems. We propose a semi-automatic method for annotating the shape and pose of hands and objects in the collected videos, significantly reducing the annotation time and cost compared to manual labeling. With this system, we captured a video dataset of humans performing various single- and dual-hand manipulation tasks, including simple pick-and-place actions, handovers between hands, and using objects according to their affordance. This dataset can serve as human demonstrations for research in embodied AI and robot manipulation. Our capture setup and annotation framework will be made available to the community for reconstructing 3D shapes of objects and human hands, as well as tracking their poses in videos. 

   more » « less
2. Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos

   Chang, Matthew; Prakash, Aditya; Gupta, Saurabh (December 2023, Neural Information Processing Systems)

   The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos. 

   more » « less
3. Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on Youtube

   Aravind Sivakumar; Kenneth Shaw; Deepak Pathak (July 2022, Robotics: Science and Systems)

   We build a system that enables any human to control a robot hand and arm, simply by demonstrating motions with their own hand. The robot observes the human operator via a single RGB camera and imitates their actions in real-time. Human hands and robot hands differ in shape, size, and joint structure, and performing this translation from a single uncalibrated camera is a highly underconstrained problem. Moreover, the retargeted trajectories must effectively execute tasks on a physical robot, which requires them to be temporally smooth and free of self-collisions. Our key insight is that while paired human-robot correspondence data is expensive to collect, the internet contains a massive corpus of rich and diverse human hand videos. We leverage this data to train a system that understands human hands and retargets a human video stream into a robot hand-arm trajectory that is smooth, swift, safe, and semantically similar to the guiding demonstration. We demonstrate that it enables previously untrained people to teleoperate a robot on various dexterous manipulation tasks. Our low-cost, glove-free, marker-free remote teleoperation system makes robot teaching more accessible and we hope that it can aid robots that learn to act autonomously in the real world. 

   more » « less
4. RAM-Hand: Robust Acoustic Multi-Hand Pose Reconstruction Using a Microphone Array

   https://doi.org/10.1145/3715014.3722055  

   Wang, Shiyang; Pu, Henglin; Cao, Qiming; Jiang, Wenjun; Wang, Xingchen; Liu, Tianci; Jiang, Zhengxin; Xue, Hongfei; Su, Lu (May 2025, ACM)

   Using 3D hand poses as the input of user interfaces can enable many novel human-computer interaction applications. However, conven- tional solutions for precisely reconstructing the hand poses are either vision-based, which are compute-intensive and may cause privacy issues, or wearable devices-based, which are intrusive to users. In this paper, we propose RAM-Hand, a Robust Acoustic 3D Multi-Hand pose reconstruction system built on a microphone array. Our RAM-Hand system can support multiple hands and is designed to be highly adaptable to new scenarios even when training data is limited. Specifically, it should robustly accommo- date variations in environment, subject, and hand positions. To achieve this, on one hand, we propose a customized signal process- ing pipeline to segment multiple hands’ reflections and extract the features corresponding to each hand, then feed those features into a transformer-based neural network for precise pose reconstruc- tion. On the other hand, to tackle the challenge that the training data is limited, we propose a series of data augmentation methods to generate virtual training data, and utilize contrastive learning to ensure our model behaves well on new subjects. We conduct extensive experiments on a real-world microphone array testbed to evaluate the performance of the proposed system. The results show that our RAM-Hand system can localize each hand joint with an average error of 10.71mm, handle multiple hands, and generalize well to the above mentioned new scenarios. 

   more » « less
5. Towards Smartphone-based 3D Hand Pose Reconstruction Using Acoustic Signals

   https://doi.org/10.1145/3677122  

   Wang, Shiyang; Wang, Xingchen; Jiang, Wenjun; Miao, Chenglin; Cao, Qiming; Wang, Haoyu; Sun, Ke; Xue, Hongfei; Su, Lu (September 2024, ACM Transactions on Sensor Networks)

   Accurately reconstructing 3D hand poses is a pivotal element for numerous Human-Computer Interaction applications. In this work, we propose SonicHand, the first smartphone-based 3D hand pose reconstruction system using purely inaudible acoustic signals. SonicHand incorporates signal processing techniques and a deep learning framework to address a series of challenges. First, it encodes the topological information of the hand skeleton as prior knowledge and utilizes a deep learning model to realistically and smoothly reconstruct the hand poses. Second, the system employs adversarial training to enhance the generalization ability of our system to be deployed in a new environment or for a new user. Third, we adopt a hand tracking method based on channel impulse response estimation. It enables our system to handle the scenario where the hand performs gestures while moving arbitrarily as a whole. We conduct extensive experiments on a smartphone testbed to demonstrate the effectiveness and robustness of our system from various dimensions. The experiments involve 10 subjects performing up to 12 different hand gestures in three distinctive environments. When the phone is held in one of the user’s hands, the proposed system can track joints with an average error of 18.64 mm. 

   more » « less