More Like this

1. Object Motion Guided Human Motion Synthesis

   https://doi.org/10.1145/3618333  

   Li, Jiaman; Wu, Jiajun; Liu, C Karen (December 2023, ACM Transactions on Graphics)

   Modeling human behaviors in contextual environments has a wide range of applications in character animation, embodied AI, VR/AR, and robotics. In real-world scenarios, humans frequently interact with the environment and manipulate various objects to complete daily tasks. In this work, we study the problem of full-body human motion synthesis for the manipulation of large-sized objects. We propose Object MOtion guided human MOtion synthesis (OMOMO), a conditional diffusion framework that can generate full-body manipulation behaviors from only the object motion. Since naively applying diffusion models fails to precisely enforce contact constraints between the hands and the object, OMOMO learns two separate denoising processes to first predict hand positions from object motion and subsequently synthesize full-body poses based on the predicted hand positions. By employing the hand positions as an intermediate representation between the two denoising processes, we can explicitly enforce contact constraints, resulting in more physically plausible manipulation motions. With the learned model, we develop a novel system that captures full-body human manipulation motions by simply attaching a smartphone to the object being manipulated. Through extensive experiments, we demonstrate the effectiveness of our proposed pipeline and its ability to generalize to unseen objects. Additionally, as high-quality human-object interaction datasets are scarce, we collect a large-scale dataset consisting of 3D object geometry, object motion, and human motion. Our dataset contains human-object interaction motion for 15 objects, with a total duration of approximately 10 hours. 

   more » « less
2. Collective Motion in Human Crowds

   https://doi.org/10.1177/0963721417746743  

   Warren, William H. (July 2018, Current Directions in Psychological Science)
3. HuMoR: 3D Human Motion Model for Robust Pose EstimationHuMoR: 3D Human Motion Model for Robust Pose Estimation

   Rempe, Davis and (January 2021, International Conference on Computer Vision (ICCV))

   We introduce HuMoR: a 3D Human Motion Model for Robust Estimation of temporal pose and shape. Though substantial progress has been made in estimating 3D human motion and shape from dynamic observations, recovering plausible pose sequences in the presence of noise and occlusions remains a challenge. For this purpose, we propose an expressive generative model in the form of a conditional variational autoencoder, which learns a distribution of the change in pose at each step of a motion sequence. Furthermore, we introduce a flexible optimization-based approach that leverages HuMoR as a motion prior to robustly estimate plausible pose and shape from ambiguous observations. Through extensive evaluations, we demonstrate that our model generalizes to diverse motions and body shapes after training on a large motion capture dataset, and enables motion reconstruction from multiple input modalities including 3D keypoints and RGB(-D) videos. 

   more » « less
4. Integrating Uncertainty-Aware Human Motion Prediction Into Graph-Based Manipulator Motion Planning

   https://doi.org/10.1109/TMECH.2024.3402682  

   Liu, Wansong; Eltouny, Kareem; Tian, Sibo; Liang, Xiao; Zheng, Minghui (August 2024, IEEE/ASME Transactions on Mechatronics)
5. Dynamic Motion Representation for Human Action Recognition

   https://doi.org/10.1109/WACV45572.2020.9093500  

   Asghari-Esfeden, Sadjad; Sznaier, Mario; Camps, Octavia (March 2020, 2020 IEEE Winter Conference on Applications of Computer Vision (WACV))

   Despite the advances in Human Activity Recognition, the ability to exploit the dynamics of human body motion in videos has yet to be achieved. In numerous recent works, re- searchers have used appearance and motion as independent inputs to infer the action that is taking place in a specific video. In this paper, we highlight that while using a novel representation of human body motion, we can benefit from appearance and motion simultaneously. As a result, bet- ter performance of action recognition can be achieved. We start with a pose estimator to extract the location and heat- map of body joints in each frame. We use a dynamic encoder to generate a fixed size representation from these body joint heat-maps. Our experimental results show that training a convolutional neural network with the dynamic motion representation outperforms state-of-the-art action recognition models. By modeling distinguishable activities as distinct dynamical systems and with the help of two stream net- works, we obtain the best performance on HMDB, JHMDB, UCF-101, and AVA datasets. 

   more » « less