An official website of the United States government
We tackle the novel problem of predicting 3D hand motion and contact maps (or Interaction Trajectories) given a single RGB view, action text, and a 3D contact point on the object as input. Our approach consists of (1) Interaction Codebook: a VQVAE model to learn a latent codebook of hand poses and contact points, effectively tokenizing interaction trajectories, (2) Interaction Predictor: a transformer-decoder module to predict the interaction trajectory from test time inputs by using an indexer module to retrieve a latent affordance from the learned codebook. To train our model, we develop a data engine that extracts 3D hand poses and contact trajectories from the diverse HoloAssist dataset. We evaluate our model on a benchmark that is 2.5-10X larger than existing works, in terms of diversity of objects and interactions observed, and test for generalization of the model across object categories, action categories, tasks, and scenes. Experimental results show the effectiveness of our approach over transformer & diffusion baselines across all settings.
more »
« less
Contact Edit: Artist Tools for Intuitive Modeling of Hand-Object Interactions
Posing high-contact interactions is challenging and time-consuming, with hand-object interactions being especially difficult due to the large number of degrees of freedom (DOF) of the hand and the fact that humans are experts at judging hand poses. This paper addresses this challenge by elevating contact areas to first-class primitives. We provideend-to-end art-directable(EAD) tools to model interactions based on contact areas, directly manipulate contact areas, and compute corresponding poses automatically. To make these operations intuitive and fast, we present a novel axis-based contact model that supports real-time approximately isometry-preserving operations on triangulated surfaces, permits movement between surfaces, and is both robust and scalable to large areas. We show that use of our contact model facilitates high quality posing even for unconstrained, high-DOF custom rigs intended for traditional keyframe-based animation pipelines. We additionally evaluate our approach with comparisons to prior art, ablation studies, user studies, qualitative assessments, and extensions to full-body interaction.
more »
« less
- Award ID(s):
- 1925130
- PAR ID:
- 10470340
- Publisher / Repository:
- ACM Transactions on Graphics
- Date Published:
- Journal Name:
- ACM Transactions on Graphics
- Volume:
- 42
- Issue:
- 4
- ISSN:
- 0730-0301
- Page Range / eLocation ID:
- 1 to 20
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Past decades featured significant advancements in additive and micromanufacturing that facilitated the creation of functional patterned surfaces with impressive spatial resolutions. However, these techniques are expensive and require a considerable amount of time and energy, and hence lack scalability to practical surfaces. Recent techniques employing spinodal decomposition and instabilities amplified via centrifugal acceleration offer viable and cheaper alternatives. The patterns created by those techniques, however, vary randomly in geometry. When interfacing those patterned surfaces with other components and under self-contact scenarios, geometric variations lead to stress concentration and abrupt failure around the contact. In this study, we investigate numerically real contact areas, contact tractions, and stress concentration. We generate patterned surfaces in congruence with actual surfaces created by those techniques. Then, we conduct normal-contact analyses of those surfaces boundary element method (BEM) under nominal mean pressures ranging from 0.001E* toE*, whereE* is the contact modulus. We record real contact areas and stress concentration as a function of nominal mean pressures. We compare these values with the analytical solutions from sinusoidally-patterned and randomly rough surfaces. Randomness in pattern geometry is primarily influenced by the processing parameters such as the degree of anisotropy in spinodal decomposition and acceleration in amplified instabilities. To understand the influence of the processing parameters, we perform a parametric study. We find isotropic spinodal decomposition creates patterns that deliver contact area and traction distributions similar to randomly rough surfaces, and lead to high-stress concentrations. Such high-stress concentrations are expected to occur under self-contact loading scenarios, and thus can explain the compromised resilience and strength in recently-proposed spinodal metamaterials. For patterned surfaces created by amplified instabilities, high-stress concentrations are obtained for the surfaces created at high accelerations. At high accelerations, increased elastic instabilities and stochastic growth result in a more skewed and broader distribution in heights. Therefore, high-stress concentrations are inevitable. To account for combined loading scenarios, we conduct additional simulations on the same surface patterns with frictional pre-sliding contacts. We find the frictional tractions play a secondary role in stress concentrations where the primary factor is the processing parameters determining the degree of randomness in pattern geometry.more » « less
-
Ground contact modeling for multilegged locomotion is challenging due to the possibility of multiple slipping legs. To understand the interplay of contact forces among multiple legs, we integrated a robot with six high-precision 6 degree-of-freedom (DoF) force-torque sensors, and measured the wrenches (forces and torques) produced in practice. Here, we present an in situ calibration procedure for simultaneously measuring all foot contact wrenches of a hexapod using 6-DoF load cells installed at the hips. We characterized transducer offset, leg gravity offset, and the wrench transformation error in our calibration model. Our calibration reduced the root-mean-square-error (RSME) by 63% for forces and 90% for torques in the residuals of the robot standing in different poses, compared with naive constant offset removal.more » « less
-
Across a plethora of social situations, we touch others in natural and intuitive ways to share thoughts and emotions, such as tapping to get one’s attention or caressing to soothe one’s anxiety. A deeper understanding of these human-to-human interactions will require, in part, the precise measurement of skin-to-skin physical contact. Among prior efforts, each measurement approach exhibits certain constraints, e.g., motion trackers do not capture the precise shape of skin surfaces, while pressure sensors impede skin-to-skin contact. In contrast, this work develops an interference-free 3D visual tracking system using a depth camera to measure the contact attributes between the bare hand of a toucher and the forearm of a receiver. The toucher’s hand is tracked as a posed and positioned mesh by fitting a hand model to detected 3D hand joints, whereas a receiver’s forearm is extracted as a 3D surface updated upon repeated skin contact. Based on a contact model involving point clouds, the spatiotemporal changes of hand-to-forearm contact are decomposed as six, high-resolution, time-series contact attributes, i.e., contact area, indentation depth, absolute velocity, and three orthogonal velocity components, together with contact duration. To examine the system’s capabilities and limitations, two types of experiments were performed. First, to evaluate its ability to discern human touches, one person delivered cued social messages, e.g., happiness, anger, sympathy, to another person using their preferred gestures. The results indicated that messages and gestures, as well as the identities of the touchers, were readily discerned from their contact attributes. Second, the system’s spatiotemporal accuracy was validated against measurements from independent devices, including an electromagnetic motion tracker, sensorized pressure mat, and laser displacement sensor. While validated here in the context of social communication, this system is extendable to human touch interactions such as maternal care of infants and massage therapy.more » « less
-
Abstract Field studies of cleaning mutualisms use a variety of methods to quantify behavioral dynamics. Studies in marine systems typically utilize data recorded by human observers on scuba or snorkel or via remote underwater video. The effects of these different methods on cleaner–client behaviors have not been rigorously assessed. We quantified cleaner–client interactions at 13 bluestreak cleaner wrasse (Labroides dimidiatus) cleaning stations in Moorea, French Polynesia using hand‐held and remote videos. We found that cleaning, cheating, and client posing rates, cleaning duration, and client species richness were all greater in the remote than in the hand‐held videos, suggesting that human presence disrupts cleaning interactions by inducing antipredator responses among clients. Some metrics, such as the ratio of cleaner chasing to cleaning behavior and the cleaners' benthic feeding rate, were higher for the hand‐held than the remote videos, possibly due to limited access of cleaners to clients in the presence of humans. Other metrics, such as cleaner and client chasing rates, the ratio of cleaning to cheating behaviors, and the duration of cleaner chases, did not differ between video types. Finally, piscivorous clients were far more abundant in the remote than the hand‐held videos, suggesting that piscivores are particularly sensitive to human presence, likely because they are targeted by fishers. Overall, our study suggests that human presence can bias studies of cleaning behavior and cleaner–client interactions, and that remote cameras should be used to conduct behavioral studies. These potential biases should be considered when interpreting existing behavioral data.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1145/3592117
