Generating animations from natural language descriptions is a first step for movie script visualization [11,20] which can later be stitched together while maintaining coreferences in the story-line [38]. These language grounded animations can also be useful in cases like virtual human animation [30,7,6], robot motion and task planning [16,2].
An animation consists of a sequence of poses, which can be represented by positions of different joints in the body such as Root (base of spine), head, shoulder, wrist, knee and many more.
Pose forecasting conditioned on natural language has 3 major challenges. First, pose and natural language are very different modalities. The model needs a joint space where both natural language sentences and poses can be mapped. The model should also be able to decode animations from this embedding space. Second, different Figure 2: Overview of our proposed model Joint Language-to-Pose (or JL2P). Language and pose are mapped to a joint embedding space Z, which can now be used by a trained pose decoder q d to generate a pose sequence. At train time both p e and q e are used to create the joint embedding using a training curriculum. But at inference time z ∈ Z is encoded by p e and decoded by q d , giving us a model which can generate a animation (or sequence of poses) from a free form description (or language). words of a sentence represent different qualities about the animation. Verbs and adverbs describe the action and speed/acceleration of the action; nouns and adjectives describe locations and directions respectively. The model has to map these concepts to small pose sequences and then stitch them to render convincing animations. Third, we want to see if objective metrics correlate with subjective metrics for this task as our models are trained using objective distance metrics, but the quality of generated animations can only be judged by humans.
In this paper, our two main contributions tackle the modeling challenges of pose and natural language. First, we propose a model Joint Language-to-Pose (or JL2P) that learns a joint embedding space of these two modalities. Second, we use a training curriculum to help the model emphasize more on shorter and and easier sequences first and longer and harder sequences later. Additionally, to make the training regimen robust to the outliers in the dataset, we use Smoooth L1 as the distant metric in our loss function. Through multiple objective and subjective experiments, we show that our model can generate more accurate and natural animations from natural language sentences than other data driven models.
Pose Forecasting: Data driven human pose forecasting attempts to understand the behaviours of the subject from its history of poses and generates the next sequence of poses. Short-term predictions [24] focus on modeling joint angles corresponding to hands, legs, head and torso. Long-term predictions [10,31,24] additionally model the positions of the human character to generate animations like walking, running, jumping and crawling.
While some works use different actions (such as running, kicking, and more) as conditioning variables to generate the future pose [31,18], others rely solely on the history of poses to predict what kind of motion will follow [8]. Pose forecasting for locomotion is a more commonly researched topic, where models decide where and when to run/walk based on low-level control parameters such as trajectory and terrain [13]. Task based locomotion (such as writing on a whiteboard, moving a box, and sitting on a box) add the nuances of transitioning from one task to another, but pose generation is based on task-specific footstep plans that act as motion templates [1].
All these approaches are either action specific, or require a set of low-level control parameters to forecast future pose. In this work, we aim replace low-level control parameters with high-level control parameters (e.g. natural language) to control actions and their speed and direction for the generated pose.
Image or Speech conditioned pose forecasting: Images with a human can act as a context to forecast what comes next. Chao et. al. [5] use one image frame to predict the next few poses. These generated poses can now be used to aid the generation of a video [35] or a sequence of images [19]. An image, a high-level control parameter, has action information for pose generation, but it does not provide a fine-grained control on the speed and acceleration of the motion trajectory.
Speech can also be used to control animations of virtual characters. Taylor et. al. [32] use a data driven approach to model facial animation, while upper body pose forecasting conditioned on speech inputs has been tackled by Takeuchi et. al. [30]. But, these pose sequences model the non-verbal behaviours (such as head nods, pose switches, hand waving and so on) of the character and do not offer fine-grained control over the characters next movements.
Language conditioned pose forecasting: Natural language sentences consists of verbs describing the actions, adverbs describing the speed/acceleration of the action, and nouns with adjectives to describe the direction or target. This information can help provide a more fine-grained control over pose generations compared to image or speech.
Statistical models [29,28] which use bigram models for natural language have been trained to encode motion sequences from sentences. Ahn et. al. [2] use around 2100 hours of youtube videos with annotated text descriptions to train a pose generation model. Pose sequences extracted from videos have limited translation and occluded lower bodies, hence their model only predicts the upper body with a static Root joint. Some works use 3D motion capture data instead [26,34].
Human motions generally have translation of the Root joint, hence forecasting trajectory is important to get natural looking animations. Lin et. al [17] generates pose of all the joints of the body by pretraining a pose2pose autoencoder model before mapping language embeddings on the learned pose space. But the embedding space is not learned jointly [23] which may limit the generative powers of the pose decoder. In contrast, our proposed approach learns a joint embedding space of language and pose using a curriculum learning training regime.
As an example, consider a natural language sentence which describes a human's motion: "A person walks in a circle". The goal of this cross-modal language-to-pose translation task is to generate an animation representing the sentence; i.e. an animation that shows a person following a trajectory of a circle with a walking motion (see figure 1).
Formally, given a sentence, represented by an N-sized sequence of words
that are coherent with the semantics in the sentence. x i ∈ R K is the i th word vector with dimension K. y t ∈ R J×3 is the pose matrix at time t. Rows of y t represent joints of the skeleton and columns are the xyz-coordinates of each joint. Tensors X and Y are elements of sets X and Y respectively.
Modeling language-to-pose is done by training a model f : R K×N -→ R J×3×T to predict a pose sequence Ŷ1:T
where Θ are trainable parameters of the model f .
Language-to-pose models should be able to grasp nuanced concepts like speed, direction of motion and the kind of actions from the language and translate them to pose sequences (or animations). This requires the model to learn a multimodal joint space of language and pose. In doing so, it should also be able to generate sequences that are deemed correlated to the sentence by humans.
To achieve that objective, we propose Joint Languageto-Pose (or JL2P) model to learn the joint embedding space. Given an input sentence, an animation can be sampled from this model at inference stage.
In this section, a joint embedding space of language and pose is formalized. This is followed by an algorithm to train for the joint embedding space and a discussion on the practical edge cases at inference time for our Joint Language-to-Pose model.
To learn a joint embedding space of language and pose, the sentence X 1:N and pose Y 1:T are first mapped to a latent representation using a sentence encoder p e (X 1:N ; Φ e ) and a pose encoder q e (Y 1:T ; Ψ e ) respectively. These estimate the latent representation or embeddings z x and z y respectively in the embedding space Z ⊂ R h ,
z x , z y should lie close to each other in Z as they represent the same concept. To ensure that they do lie close together, a joint translation loss is constructed (refer to Figure 2) and trained end to end with a training curriculum.
Once we have the embedding z x or z y , a pose decoder q d (; Ψ d ) is used to generate an animation from the joint embedding space Z. The output of the pose decoder must now lie close to the pose sequence Y 1:T . Hence, using X 1:N as inputs and Y 1:T as outputs, the cross-modal translation loss is defined as,
and using Y 1:T as inputs and Y 1:T as outputs, the uni-modal translation (or autoencoder) loss is defined as,
where d(x, y) is a function to calculate the distance between the predicted values and ground truth of pose. Φ e , Ψ e and Ψ d are trainable parameters of the sentence encoder, pose encoder and pose decoder respectively.
Combining equations 4 and 5 we get a joint translation loss,
Jointly optimizing the loss L j pushes z x and z y closer together improving generalizability and additionally trains the pose decoder which is useful for inference from the joint embedding space. As L j is a mutivariate function in X 1:N and Y 1:T , coordinate descent [33] for optimizing the loss function is a natural choice and is described in Algorithm 1.
Cross modal pose forecasting can be a challenging task to train [5]. Starting with simpler examples before moving on to tougher ones can be beneficial to the training process [4,37,36].
The curriculum design commonly used for pose forecasting [5] is adapted for our joint model. We first optimize the model to predict 2 time steps conditioned on the complete sentence. This easy task helps the model learn very short pose sequences like leg motions for walking, hand motions for waving and torso motions for bending. Once the loss on the validation set starts increasing, we move on to the next stage in the curriculum. The model is now given twice the amount of poses for prediction. The complexity of the task is increased in every stage till the maximum time-steps (T ) of prediction is reached. We describe the complete training process in Algorithm 1.
Algorithm 1 Learning language-pose joint embedding
for all X 1:N , Y 1:t ∈ X train , Y train do z ← q e (Y 1:t ; Ψ e ) //Encoder MaxValLoss ← inf
For the distance metric d(x, y) in Equation 4, 5 and 6, Smooth L1 loss (similar to Huber Loss [15]) is used which is defined as, SmoothL1(x, y) = 0.5(x -y)foot_1 for |x -y| < 1 |x -y| -0.5 otherwise (7) In contrast, Lin et. al. [17] uses L2 loss for d(x, y). L2 loss is more sensitive to outliers than L1 loss due to its linearly proportional gradient with respect to the error, while L1 loss has a constant gradient of 1 or -1. But L1 Loss can become unstable when |x -y| ≈ 0, due to oscillating gradients between 1 and -1. On the other hand, Smooth L1 is continuous and smooth near 0 and more generally for all x, y ∈ R, hence it is more stable than L1 as a loss function.
Joint language to pose modeling can be broken down into three core challenges, 1. Prediction Accuracy by Joint Space: How accurate is pose prediction from the joint embedding ?
2. Human Judgment: Which of the generated animation is more representative of the input sentence? Does the subjective evaluation correlate with the results from the objective evaluations?
3. Modeling nuanced language concepts: Is the model able to map nuanced concepts such as speed, direction and action in the generated animations?
Experiments are designed to evaluate these challenges of language grounded pose forecasting.
In the following subsections, the dataset and its preprocessing is briefly which is followed by the evaluation metrics for both objective and subjective evaluations. Finally, design choices of the encoder and decoder models are described which are used to construct the baselines in the final subsection.
Our models are trained an evaluated on KIT Motion-Language Dataset [25] which combines human motion with natural language descriptions. It consists of 3911 recordings (approximately 11.23 hours) which are re-targeted to a kinematic model of a human skeleton with 50 DoFs (6 DoF for the Root joint's orientation and position, while remaining 44 DoFs for arms legs, head and torso). The dataset also consists of 6278 English sentences (approximately 8 words per sentence) describing the recordings. This is more than the number of recordings as each recording has one or more descriptions which are annotated by human volunteers. We use 20% of the data as a randomly sampled held-out set for evaluating all models.
There is wide variety of motions in this dataset ranging from locomotion (e.g. walking, running, jogging), performing (e.g. playing violin/guitar), and gesticulation (e.g. waving). Many recording have adjectives to further describe the motion like speed (e.g. fast and slow), direction (left, right and forward), and number for periodic motions (e.g. walk 4 steps).
We use the pre-processing steps used in Holden et. al. [14]. All the frames of the motion are transformed such that body always faces the Z-axis. Joint rotation angles are transformed to 3D positions is the skeleton's local frame of reference with Root as the origin. Root's position on XZplane and orientation along Y-axis is represented as velocity instead of absolute values.
Motion sequences are then sub-sampled to a frequency of 12.5 Hz down from 100Hz. This is low enough to bring enough variance between 2 time-steps for the decoder to train for a regression task, while not compromising on the human's perception of the animation [5].
For pose encoder (q e ) a network of Gated Recurrent Units (GRUs) [9] is used in our model JL2P. The pose decoder (q d ) is the same except it has residual connection from the input to the output layer. This is similar to the pose decoder in Lin et. al. [17], except an extra layer to predict the trajectory (or Trajectory Predictor) is discarded in our model.
For langauge encoder (p e ), a network of Long-Short Term Memory Units (LSTMs) [12] is used. Each token of the sentence is converted into a distributional embedding using a pre-trained Word2Vec model [22]. 1
There has been limited work done in the domain of datadriven cross-modal translation from natural language descriptions to pose sequence generation. The closest work to our proposed approach is by Lin et. al. [17] 2 . As mentioned in Section 2, their model does not follow a training curriculum and uses L2 loss as the loss function. Thier model also maps the language embeddings to an existing embedding space of poses instead of jointly learning it.
We also compare our model JL2P (see Section 4) with three ablations derived from itself. These ablations study the 3 main components of the model, joint embedding space, curriculum learning and Smooth L1 loss: • JL2P w/o L1 -L2 loss is used instead of Smooth L1 loss as the distance metric d(x, y).
• JL2P w/o Joint Emb. -Instead of joint training as described in Section 4.2, autoencoder loss L u minimized first followed by optimization of the cross-translation loss L c .
All models are evaluated on the held-out set with a metric Average Position Error (APE). Given a particular joint j, it can be denoted as APE(j),
where y t [j] is the true location and ŷt [j] ∈ Y is the predicted location of joint j at time t Another metric, Probability of Correct Keypoints (PCK) [3,27], is also used as an evaluation metric.
Joint language to pose generation is a subjective task, hence a human's subjective judgment on the quality of prediction is an important metric for this task.
To achieve this, we design a user study which asked human annotators to rank two videos generated by 2 different models but with same sentence as the input. One of the videos is generated by Lin et. al. and the other is either ground truth or generated by JL2P , JL2P w/o Curriculum, JL2P w/o Joint Emb., or JL2P L1. The annotators answer the following question for each pair of videos, Which of the 2 generated animations is better described by "<sentence>"?. To ensure that annotators spend enough time to decide, any annotations which took less than 20 seconds 3 were rejected. This study subjectively evaluates the preference of humans for generated animations by different models.
In this section we first use objective measures and then conduct a user study to get a subjective evaluation. Finally, we probe some qualitative examples to understand the effectiveness of the model in tackling the core challenges described in Section 5.
JL2P demonstrates at least a 9% improvement over Lin et. al. (see Table 1) for all joints. The maximum improvement around 15% is seen in the Root joint. Errors in Root prediction can lead to a "sliding-effect" of the feet; when the generation is translating faster than the frequency of the feet. Improvements in APE scores for long-term prediction, especially for Root, can help get rid of these artifacts in the generated animation.
When compared to its variants, JL2P loses maximum APE value when it is trained without curriculum (or JL2P w/o Curriculum). As discussed in Section 4.3, learning to predict shorter sequences before moving on to longer ones proves beneficial for pose generation. APE scores go down by 4%, if L2 loss is used instead of Smooth L1. In an output space as diverse as pose sequences, it becomes important to ignore outliers which may drive model to overfit. APE scores go down only by 1%, if the embedding space is not trained with the joint loss L j APE values across time for JL2P of different parts of the body (Root, Legs, Arms, Torso and Head) are plotted in Figure 6. Root's APE scores have the fastest rate of increase, followed by Arms, Legs and then Head, Torso. Two out of three coordinates of Root are represented as velocity which accumulates errors when integrated back to absolute positions; this is probably a contributing factor to the rapid increase of prediction error over time.
Our final objective metric is PCK. PCK values (for 35≤ σ ≤55) on generated animations are compared among JL2P , its variants and Lin et. al. in Figure 7. JL2P and its ablations show a consistent improvement over Lin et. al. which further strengthen the claim about the prediction accuracy by our model's joint space.
Human judgment is quantified by preference scores in Figure 5. Human preference of all our baseline models and ground truth are compared against Lin et. al. animations. JL2P has a preference of 75% which is shy of ground truth by 10%. Preference scores consistently drop with all the other variants of JL2P . JL2P w/o Joint Emb. has the lowest preference score of 60% when ranked against Lin et. al. . It is still more preferred than Lin et. al. but far more unlikely to be picked when pitted against JL2P . This is an interesting change in trend, as removing joint loss from JL2P did not affect the objective scores significantly, but have lowered its hu- man preference by a significant fraction. This leads us to conclude that objective metrics are not enough to judge the performance of a model. Instead a combination of human judgment and objective metrics is necessary for evaluating pose generation models.
Root joint decides the trajectory of the animation which is crucial for translating concepts like speed (e.g. fast, and slow), direction (e.g. left, right, forward and backward) from natural language to animation. We plot the trajectories generated by JL2P, ground truth and Lin et. al. for different sentences in Figure 3.
Modeling direction: Animations' trajectory for these sentences for JL2P is similar to that of the ground truth trajectories. In contrast, Lin et. al.'s trajectories tend to be semantically incorrect and have a slightly curved forward motion for these sentences.
Modeling speed: In the sentence, "A person runs very fast forward", JL2P is able to understand that the animation has to move faster. It is able to walk approximately the same distance as the ground truth in the same amount of time, hence has the same speed. In contrast, even though Lin et. al.'s motion is in the forward direction, it is not able to maintain the same speed as required by the sentence.
Modeling actions: In figure 4, we plot animations generated by a diverse set of sentences. JL2P is able to understand the action from the sentences, and is able to generate an animation corresponding to the action. JL2P is able to handle many actions ranging from kneeling (with complex leg motions) to jogging (with periodic hand and leg motion).
We show, via qualitative examples, that our model JL2P is able to model nuanced language concepts which are then reproduced in the animations generated at inference time.
In this paper, we proposed a neural architecture called Joint Language-to-Pose (or JL2P), which integrates language and pose to learn a joint embedding space in an endto-end training paradigm. This embedding space can now be used to generate animations conditioned on an input description. We also proposed the use of curriculum learning approach which forces the model to generate shorter sequences before moving on to longer ones. We evaluated our proposed model on a parallel corpus of 3D pose data and human-annotated sentences with objective metrics to measure prediction accuracy, as well as with a user study to measure human judgment. Our results confirm that our approach, to learn a joint embedding in a curriculum learning paradigm by JL2P, was able to generate more accurate animations and are deemed more visually represented by humans than the state-of-the-art model.
We also train a variant of the model with BERT as the language encoder, but it did not show any significant improvements.
As we could not find code or pre-trained models for this work, we use our own implementation and training on the same data as all other baselines
each video is 8 seconds long at an average. We set a threshold of 20 seconds to give annotators
seconds to make their decision.