More Like this

1. Object Motion Guided Human Motion Synthesis

   https://doi.org/10.1145/3618333  

   Li, Jiaman; Wu, Jiajun; Liu, C Karen (December 2023, ACM Transactions on Graphics)

   Modeling human behaviors in contextual environments has a wide range of applications in character animation, embodied AI, VR/AR, and robotics. In real-world scenarios, humans frequently interact with the environment and manipulate various objects to complete daily tasks. In this work, we study the problem of full-body human motion synthesis for the manipulation of large-sized objects. We propose Object MOtion guided human MOtion synthesis (OMOMO), a conditional diffusion framework that can generate full-body manipulation behaviors from only the object motion. Since naively applying diffusion models fails to precisely enforce contact constraints between the hands and the object, OMOMO learns two separate denoising processes to first predict hand positions from object motion and subsequently synthesize full-body poses based on the predicted hand positions. By employing the hand positions as an intermediate representation between the two denoising processes, we can explicitly enforce contact constraints, resulting in more physically plausible manipulation motions. With the learned model, we develop a novel system that captures full-body human manipulation motions by simply attaching a smartphone to the object being manipulated. Through extensive experiments, we demonstrate the effectiveness of our proposed pipeline and its ability to generalize to unseen objects. Additionally, as high-quality human-object interaction datasets are scarce, we collect a large-scale dataset consisting of 3D object geometry, object motion, and human motion. Our dataset contains human-object interaction motion for 15 objects, with a total duration of approximately 10 hours. 

   more » « less
2. Automated Design of Simple and Robust Manipulators for Dexterous In-Hand Manipulation Tasks using Evolutionary Strategies

   https://doi.org/10.1109/Humanoids43949.2019.9035055  

   Meixner, Andre; Hazard, Christopher; Pollard, Nancy (October 2019, 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids))

   null (Ed.)

   In spite of substantial progress, robust and dexterous in-hand manipulation remains a robotics grand challenge. Recent research has shown that optimization of robot hand morphology for specific tasks can result in custom hand designs that are low-cost, easy to maintain, and highly capable. However, the resulting manipulation strategies may not be very robust or generalizable in real-world situations. This paper shows that robustness can be improved dramatically by optimizing controls instead of contact force / trajectories and by considering uncertainty explicitly during the optimization process. We present a evolutionary algorithm based pipeline for co-optimizing hand morphology and control strategy over families of problems and initial states in order to achieve robust in-hand manipulation. We demonstrate that this approach produces robust results which utilize all surfaces of the hand and surprising dynamic motions. We showcase the advantage of optimizing joint limit values to create robust designs. Furthermore, we demonstrate that our approach is complementary to trajectory optimization based approaches and can be utilized to improve robustness of such results as well as to create custom hand designs from scratch. Results are shown for repositioning and reorienting diverse objects relative to the palm of the hand. 

   more » « less
3. Manipulation for self-Identification, and self-Identification for better manipulation

   https://doi.org/10.1126/scirobotics.abe1321  

   Hang, Kaiyu; Bircher, Walter G.; Morgan, Andrew S.; Dollar, Aaron M. (May 2021, Science Robotics)

   The process of modeling a series of hand-object parameters is crucial for precise and controllable robotic in-hand manipulation because it enables the mapping from the hand’s actuation input to the object’s motion to be obtained. Without assuming that most of these model parameters are known a priori or can be easily estimated by sensors, we focus on equipping robots with the ability to actively self-identify necessary model parameters using minimal sensing. Here, we derive algorithms, on the basis of the concept of virtual linkage-based representations (VLRs), to self-identify the underlying mechanics of hand-object systems via exploratory manipulation actions and probabilistic reasoning and, in turn, show that the self-identified VLR can enable the control of precise in-hand manipulation. To validate our framework, we instantiated the proposed system on a Yale Model O hand without joint encoders or tactile sensors. The passive adaptability of the underactuated hand greatly facilitates the self-identification process, because they naturally secure stable hand-object interactions during random exploration. Relying solely on an in-hand camera, our system can effectively self-identify the VLRs, even when some fingers are replaced with novel designs. In addition, we show in-hand manipulation applications of handwriting, marble maze playing, and cup stacking to demonstrate the effectiveness of the VLR in precise in-hand manipulation control. 

   more » « less
4. Path Planning for Within-Hand Manipulation over Learned Representations of Safe States

   Calli, Berk; Kimmel, Andrew; Hang, Kaiyu; Bekris, Kostas; Dollar, Aaron (October 2018, International Symposium on Experimental Robotics (ISER))

   This work proposes a framework for tracking a desired path of an object held by an adaptive hand via within-hand manipulation. Such underactuated hands are able to passively achieve stable contacts with objects. Combined with vision-based control and data-driven state estimation process, they can solve tasks without accurate hand-object models or multi-modal sensory feedback. In particular, a data-driven regression process is used here to estimate the probability of dropping the object for given manipulation states. Then, an optimization-based planner aims to track the desired path while avoiding states that are above a threshold probability of dropping the object. The optimized cost function, based on the principle of Dynamic-Time Warping (DTW), seeks to minimize the area between the desired and the followed path. By adapting the threshold for the probability of dropping the object, the framework can handle objects of different weights without retraining. Experiments involving writing letters with a marker, as well as tracing randomized paths, were conducted on the Yale Model T-42 hand. Results indicate that the framework successfully avoids undesirable states, while minimizing the proposed cost function, thereby producing object paths for within-hand manipulation that closely match the target ones. 

   more » « less
5. Waypoint-Based Reinforcement Learning for Robot Manipulation Tasks

   https://doi.org/10.1109/IROS58592.2024.10802681  

   Mehta, Shaunak A; Habibian, Soheil; Losey, Dylan P (October 2024, Proceedings of the International Conference on Intelligent Robots and Systems)

   Robot arms should be able to learn new tasks. One framework here is reinforcement learning, where the robot is given a reward function that encodes the task, and the robot autonomously learns actions to maximize its reward. Existing approaches to reinforcement learning often frame this problem as a Markov decision process, and learn a policy (or a hierarchy of policies) to complete the task. These policies reason over hundreds of fine-grained actions that the robot arm needs to take: e.g., moving slightly to the right or rotating the end-effector a few degrees. But the manipulation tasks that we want robots to perform can often be broken down into a small number of high-level motions: e.g., reaching an object or turning a handle. In this paper we therefore propose a waypoint-based approach for model-free reinforcement learning. Instead of learning a low-level policy, the robot now learns a trajectory of waypoints, and then interpolates between those waypoints using existing controllers. Our key novelty is framing this waypoint-based setting as a sequence of multi-armed bandits: each bandit problem corresponds to one waypoint along the robot’s motion. We theoretically show that an ideal solution to this reformulation has lower regret bounds than standard frameworks. We also introduce an approximate posterior sampling solution that builds the robot’s motion one waypoint at a time. Results across benchmark simulations and two real-world experiments suggest that this proposed approach learns new tasks more quickly than state-of-the-art baselines. See our website here: https://collab.me.vt.edu/rl-waypoints/ 

   more » « less