3D Human pose and shape estimation is one of the most active fields of research within the current landscape of computer vision and AI thanks to its many applications in robotics (Du and Zhang, 2014;Zimmermann et al., 2018), activity recognition (Lo Presti and La Cascia, 2016;Carbonera Luvizon et al., 2017), graphics (Boulic et al., 1997;Aitpayev and Gaber, 2020) and human-object interaction detection (Fang et al., 2018;Qi et al., 2018;Li et al., 2020b;Li et al., 2020c). Current works on 3D pose and shape estimation tend to mainly focus on the estimation of the 3D joint locations relative to the root joint, usually defined as the one closest to the shape centroid.In case of humans, defined as the pelvis joint. Due to the inherent depth and scale ambiguity of a single view, it is quite a challenge to accurately recover multi-person 3D pose and shape with absolute scales from a single RGB image. Addressing this ambiguity requires assessing cues in the image as a whole, such as scene layouts, body dimensions, and inter-person relationships. This paper aims to address the problem of estimating absolute 3D pose and shape of multiple people simultaneously from a single RGB image. Compared to the 3D pose and shape estimation problem that focuses on recovering the root-relative pose, the task addressed here additionally needs to recover the 3D translation of each person in the camera coordinate system (or sometimes called camera coordinate frame).
Estimating the absolute 3D location of each person in an image is essential for understanding humanto-human interactions (Figure 3). Since multi-person activities take place in cluttered scenes, inherent depth ambiguity and occlusions make it challenging to estimate the absolute positions of multiple individuals. We argue that body dimensions alone paint a vague picture of absolute depth. Robust estimation of global positions requires information cues over the entire image, such as geometric cues, human body sizes in the image, any occlusions which might affect the perceived sizes, layout of the entire scene etc.
Most existing methods for absolute multi-person 3D pose estimation extend the single-person approach with an added step to recover the absolute position of each detected person individually. They either use another neural network to regress the 3D translation of the person from the cropped image (Moon et al., 2019) or compute it based on the prior knowledge about the body size (Dabral et al., 2019), which ignores the global context of the whole image. While others employ complicated architecture with extensive steps, drastically slowing down inference (Lin and Lee, 2020).
In this paper, we build upon ROMP (Sun et al., 2020), a light weight, accurate end to end multiperson 3D mesh prediction network, by adding in absolute root joint depth estimation and localization head while maintaining its end to end and light weight nature. We call the revised network Absolute-ROMP. We adopt the same end to end pipeline for the task of multi-person absolute 3D mesh estimation. By leveraging depth cues from the entire scene and prior knowledge of the typical size of the human pose and body joints, we can estimate the depth of a person in a monocular image with considerably high accuracy. The target depths are discretized into a preset number of bins, in order to limit the range of predictions and thus improve the prediction performance. The range of these bins is chosen after taking prediction error mitigation and reasonable distance estimation in consideration. We employ a soft-argmax operation on the bins for improved accuracy as compared to exact bin locations and for faster convergence during training without losing precision to direct numerical regression.
We also train a CNN and a vision transformer hybrid network to predict the vertical field of view of the image, which is used to estimate the focal length. With the predicted focal length we can estimate absolute distance in camera coordinates without the need for camera intrinsic parameters. Our model uses embeddings from ResNet (He et al., 2015), which are then converted to tokens to be fed into the vision transformer (Dosovitskiy et al., 2020a). The added attention from the transformer gives our network an edge over previous work (Kocabas et al., 2021c).
Our contributions in this work are:
• We propose an absolute depth estimation head for the ROMP network using a combination loss.
• We design and train TransFocal, a network to pre-dict focal length of the image thus negating the requirement of intrinsic parameters.
• Quantitative and qualitative results show that our approach outperforms or has competitive performance to the state-of-the-art on multiple benchmark datasets under various evaluation metrics.
This paper is organizes as the follows. Section 2 discusses related work. Section 3 describes the proposed Absolute-ROMP, including the absolute depth map head and the focal length estimation network: TransFocal. Section 4 provides some key implementation details in network architectures and training/testing settings. Section 5 presents experimental results and ablation studies. Section 6 provides a few concluding remarks.
Parametric human body models allow complex 3D human mesh vertices to be encoded into low dimensional parameter vectors. These have been widely adopted since they allow regression of 3D meshes from images, such as the Skinned Multi-Person Linear Model (SMPL) (Loper et al., 2015). Various weak supervision techniques have lead to reasonable accuracy for single-person 3D mesh regression with various techniques, such as those using semantic segmentation (Xu et al., 2019), geometric prior (Kanazawa et al., 2017), motion analysis (Kanazawa et al., 2018;Kocabas et al., 2019) and 2D pose (Choi et al., 2020). The work in (Kocabas et al., 2021a) uses a part-guided attention mechanism to overcome occlusions by exploiting information about the visibility of individual body parts while leveraging information from neighboring body-parts to predict occluded parts. Whereas (Kocabas et al., 2021c) uses predicted camera calibration parameters to aid in the regression of the body mesh parameters.
For multi-person 3D pose estimation, (Mehta et al., 2017) proposes occlusion-robust pose-maps and exploits the body part association to avoid bounding box prediction. In (Benzine et al., 2021), an anchor-based one-stage model is proposed, which relies on a huge number of pre-defined anchor predictions and the positive anchor selection. To handle person-person occlusion, (Zhen et al., 2020) proposes a system that first regresses a set of 2.5D representations of body parts and then reconstructs the 3D absolute poses based on these 2.5D representations with a depthaware part association algorithm. Both (Rogez et al., 2019) and (Moon et al., 2019) employ a top-down design, which estimates the target via regression from anchor-based feature proposals.
2.3 Multi-person 3D mesh estimation (Jiang et al., 2020) proposes a network for Coherent Reconstruction of Multiple Humans (CRMH). Built on Faster-RCNN (Ren et al., 2015), they use the RoIaligned feature of each person to predict the SMPL parameters. (Zanfir et al., 2018b) estimates the 3D mesh of each person from its intermediate 3D pose estimation. (Zanfir et al., 2018a) further employs multiple scene constraints to optimize the multi-person 3D mesh results. All these methods follow a multi-stage design. The complex multi-step process requires a repeated feature extraction, which is computationally expensive. The ROMP (Sun et al., 2020), which our proposed model is based on, learns an explicit pixellevel representation with a holistic view, which improves accuracy in multi-person in-the-wild scenes.
Depth estimation from a single view suffers from inherent ambiguity. Nevertheless, several methods make remarkable advances in recent years (Li and Snavely, 2018;Lee and Kim, 2019). (Li et al., 2019) employs the use of mannequin challenge to obtain a dataset for depth estimation by employing the frozen poses and moving camera of the mannequin challenge. They generate training data using multiview stereo reconstruction and adopt a data-driven approach to recover a dense depth map. However, such a depth map lacks scale consistency and cannot reflect the real depth. (Zhen et al., 2020) proposes a system that first regresses a set of 2.5D representations of body parts and then reconstructs the 3D absolute poses based on these 2.5D representations with a depth-aware part association algorithm. (Lin and Lee, 2020) estimates the 2D human pose with heatmaps of the joints. Then these heatmaps are used as attention masks for pooling features from image regions corresponding to the target person. A skeleton-based Graph Neural Network (GNN) is used to predict the depth of each joint. (Li et al., 2020a) uses an integrated model to estimate human bounding boxes, human depths, and root-relative 3D poses simultaneously, with a coarse-to-fine architecture. All these methods either employ multi-step prediction or use large networks which slow down the inference.
Our method is able to estimate depth and regress 3D mesh of multiple people in real time with high accuracy while also regressing the shape parameters of individuals.
Recent works such as (Hold-Geoffroy et al., 2017;Kendall et al., 2015;Workman et al., 2016;Workman et al., 2015;Zhu et al., 2020) estimate camera parameters from a single image. To estimate camera rotations and fields of view, these methods train a neural network to leverage geometric cues in the image. (Workman et al., 2015) uses an AlexNet backbone to regress the horizontal field of view. Except (Workman et al., 2015), these methods discretize the continuous space of rotation into bins, casting the problem as a classification task and applying cross entropy (Workman et al., 2016) or KL-divergence (Hold-Geoffroy et al., 2017;Zhu et al., 2020) losses. (Kocabas et al., 2021c) also uses a binning technique. It trains a neural network with a bespoke-biased loss on a new collected dataset (Kocabas et al., 2021c) . However none of these methods take advantage of the latest architecture innovation in the vision space, i.e., transformer networks (Dosovitskiy et al., 2020a). (Lee et al., 2021) takes both an image and line segments as input and regresses the camera parameters based on the transformer encode-decoder architecture. The line segments, extracted from the input image using the LSD algorithm (Grompone von Gioi et al., 2010), are also mapped to geometric tokens. However, the subsequent transformer decoder aggregates both semantic and geometric tokens along with the queries for the camera parameters. In contrast, our model only employs vision transformer to encode the features from a single image and a simple MLP layer is used for decoding, thus an integration of a vision transformer and a CNN hybrid network with a combination of losses during supervision.
Absolute-ROMP is an extension of the ROMP network (Sun et al., 2020), which regresses meshes in a one-stage fashion for multiple 3D people (thus termed ROMP). We will briefly explain the working of our Absolute-ROMP and, therefore by extension, of ROMP before going into details about our addition.
Figure 1 shows overall system diagram. Absolute-ROMP employs a multi-head design with a HRNeT-32 backbone (Wang et al., 2019) and 4 head networks: At each position of the camera map, SMPL map and absolute depth map, it predicts the absolute depth of the root joint s z ( defined as the pelvis joint), camera parameters (t x ,t y , s) ( a weak-perspective camera model with translations in the x and y directions and a scale), and SMPL parameters (22×6 pose parameters and 10 shape parameters) of the person that takes the position as the center respectively.
During inference, we sample the 3D body mesh parameter results from the SMPL parameter map at the 2D body center locations parsed from the body center heatmap. We put the sampled parameters into the SMPL model to generate the 3D body meshes. we employ a weak-perspective camera model to project 3D body joints back to the 2D joints on the image plane. The reason for not employing perspective transform from the absolute coordinates for back projection is because the root depth map only predicts the absolute depth parameter s z , not translation in the x or y direction. This is done in order to preserve real time inference, but it is not sufficient for a perspective projection back to the image. Weak-perspective camera model allows orthogonal back-projection of the 3D pose to 2D on the image, facilitating training the model with in-the-wild 2D pose datasets. This helps with robustness and generalization and also allows for more accurate position estimates than the body center heatmap, as the body center heatmap has limited resolution compared to the image. We will provide more details on heatmap resolution in the next section. We employ the estimated focal length from TransFocal, the projected 2D pose and the predicted absolute depth of the root joint to get the absolute x and y coordinates of the root joint. Finally, using the 3D pose prediction from the SMPL mesh parameters, we infer the absolute location of each of the remaining joints.
From the original ROMP network (Sun et al., 2020), we maintain an output map size n × H × W , where n is the number of channels and H = W = 64, assuming that each location of the absolute depth map is the center of a human body, we estimate absolute depth of its corresponding root joint. Instead of directly regressing the numerical value of depth, we employ a binning technique in the log depth space. The binning resolution is set to 120. The range is chosen after taking prediction error mitigation and reasonable distance estimation based on all available data into consideration. Since different focal lengths of the camera affect the scales of a target person in the image, it is unrealistic to estimate the absolute depth from images taken by any arbitrary camera. 3D human datasets with absolute depth information have minimal variation in focal lengths (most datasets use the same camera for all images in each said dataset). This makes it challenging for the model to learn this variable and therefore can overfit on the focal lengths of the train-ing datasets. The same person will have different dimensions in the images with cameras with different focal lengths. Therefore, the predicted depth is normalized(divided) by the focal length. The focal length is estimated by training another network Trans-Focal, which we will detail in the next subsection.
We employ a soft-argmax operation on the bins for improved accuracy as compared to exact bin locations. Exact bins can only give integer outputs and therefore have limited precision. Whereas soft bins can output any number between bin indices depending on what number between 0 and 1 is the output of that bin. Therefore, there is minimal precision loss if any. This allows the actual prediction to be a ratio of the bins improving granularity and accuracy of the model. The bin index for the log depth space is computed as follows:
where b( d) is the bin index of the normalized depth d. N is the total number of bins and [S, E] is the range of the bins. We can state this style of binning depth map as a collection of 1D heatmaps.There are 64×64 predictions for each image. Therefore, we have a 64×64 collection of 1D heatmaps. The predicted bin values B are converted back to normalized depth as follows:
Similar to (Lin and Lee, 2020), the losses added to supervise depth learning are cross-entropy loss on the estimated bins B and L1 loss on the bin index b as follows:
(3)
To calculate L bins we have to generate ground truth bins. First, we compute bin index using equation 1.
The procedure to generate the final bins from the index is then laid out in the pseudo code of Algorithm 1.
Algorithm 1 Generate Ground truth Bins The final loss is as follows:
(5)
L abs = λ pose L pose + λ shape L shape + λ j3d L j3d + λ pa j3d L pa j3d + λ p j2d L p j2d + λ prior L prior + λ bins L bins + λ id L id where λ i represents the weight associated with the loss, L pose is the L2 loss of the pose parameters in the 3 × 3 rotation matrix format, L shape is the L2 loss of the shape parameters, L j3d is the L2 loss of the 3D joints, L pa j3d is the L2 loss of the 3D joints after Procrustes alignment, L p j2d is the L2 loss of the projected 2D joints, and L prior is the Mixture Gaussian prior loss of the SMPL parameters adopted in (Loper et al., 2015). please refer to (Sun et al., 2020) for further detail on these losses.
To estimate the focal length, we train a CNN and transformer hybrid network TransFocal. When it comes to images, convolutional neural networks by design have a stronger inductive bias compared to transformer networks (Bai et al., 2021). This results in them being able to learn to embeddings relatively quickly from a smaller subset of data. However, when trained on enough data, the vision transformer is able to outperform similar state-of-the-art CNN models (Dosovitskiy et al., 2020a). The transformer's self attention like architecture results in it having better generalization properties (Bai et al., 2021). It has been shown that combining CNN embeddings with the vision transformer results in the hybrid system performing better than larger and deeper vision transformer, with less than half the computational fine-tuning cost (Dosovitskiy et al., 2020b). Thus combining the benefits of both architectures we are able to train on lesser data and get improved accuracy. Since focal length in pixels has an unbounded range and it changes whenever one resizes images, we estimate vertical field of view (vfov) v in radians and convert it to focal length f y via:
where h is the image height in pixels. We follow (Zhu et al., 2020) and (Kocabas et al., 2021c) to assume zero camera yaw and the effective focal length values in both directions are the same, i.e., f x = f y = f . TransFocal takes the complete image as input to predict vfov which is the same for all subjects in the image of a video sequence. This means that inference has to only be performed once in order to get absolute coordinates for each frame of a video sequence. The full image contains rich cues that can facilitate transformer self attention. Vanishing points and geometric lines help the network semantically reason the vertical field of view. Similar to our absolute depth map head and (Kocabas et al., 2021c), we discretize the space of vfov v into B bins, converting the regression problem into a classification problem. Also similar to our depth map head, we aggregate the predicted probability mass using a softargmax operation. For the losses, we found in our testing that combining cross entropy loss L CE (with a smaller weight) with softargmaxbiased-L2 loss (Kocabas et al., 2021c) L agmax improves model convergence. The final loss L f oc :
In the following, we will list the implementation details of both the absolute depth map head, Absolute-ROMP, and the focal length estimation network, TransFocal, in terms of network architecture, training setting details, training datasets, and evaluation metrics.
Network Architecture: We employ HRNet-32 (Wang et al., 2019) as the backbone for Absolute-ROMP. We also maintain CoordConv (Liu et al., 2018) from ROMP to enhance the spatial information. Therefore, the backbone feature is the combination of a coordinate index map and output feature embeddings from HRNET-32. This feature set is then used as input to the four heads for complete end to end prediction. The architecture of absolute depth map is similar to the other map heads. The only difference being the output of the final 1×1 convolutional layer, with a size of 120×64×64: As stated earlier the binning resolution is set to 120 and the map size is 64×64. For details on the architecture of the map heads please refer to (Sun et al., 2020).
Setting Details: The input images are resized to 512 × 512, by keeping the same aspect ratio and padding with zeros. The size of the backbone feature is H b = W b = 128. The maximum number of detection is N = 64. The learning rate used is 5e-5. The batch size is set to 26. We adopt the Adam optimizer (Kingma and Ba, 2014) for training, and train the model until performance plateaus on validation set.
Training Datasets: The basic training datasets we used in the experiments include three 3D pose datasets: (Human3.6M (Ionescu et al., 2014), MPI-INF-3DHP (Mehta et al., 2016), MuCo-3DHP (Mehta et al., 2016)) and four in-the-wild 2D pose datasets (MS COCO (Lin et al., 2014), MPII (Andriluka et al., 2014), LSP (Johnson and Everingham, 2010), Crowdpose (Li et al., 2018)). We also use the pseudo 3D annotations from (Kolotouros et al., 2019) and the pseudo 3D labels of 2D pose datasets provided by (Joo et al., 2020). For evaluation on a 3D pose dataset 3DPW (von Marcard et al., 2018), we use the train set for fine tuning.
Evaluation Benchmarks and Metrics: We evaluate our model on the Human3.6M (Ionescu et al., 2014), Mupots (Mehta et al., 2017) and3DPW (von Marcard et al., 2018). To evaluate the 3D pose accuracy, we employ mean per joint position error (MPJPE) (Joo et al., 2015) and Procrustes-aligned MPJPE (PMPJPE). For root depth estimation we employ MRPE z (Moon et al., 2019) and PCK abs (Moon et al., 2019).
MPJPE is the average Euclidean distance between the location of real-life joints on human bodies and the location of predicted joints on 3D pose after translating the root joint ('pelvis') of estimated bodies to the groundtruth root. Procrustes-aligned MPJPE (PMPJPE) uses procrustes alignment (Luo and Hancock, 1999) (PA) to solve for translation, scale and rotation between the estimated bodies and the ground truth and thus mostly focuses on the pose error. Mean root position error(MRPE) is the mean of the euclidean distance between the estimated coordinates of the predicted absolute root and ground truth absolute root. The 3D percentage of correct absolute keypoints (3DPCK abs ) treats a joint's absolute prediction as correct if it lies within a 15cm from the ground truth joint location.
Network Architecture: The architecture is shown in Figure 2. Similar to (Yang et al., 2021), we use ResNet (He et al., 2015) as the CNN backbone. Then learnable patch embeddings are applied to patches extracted from the ResNet output.Each patch embedding's kernel size is equal to the patch size, which means that the input sequence is obtained by simply flattening the spatial dimensions of the ResNet features and projecting to the Transformer dimension. Since the image is not converted into patches directly, the original physical meaning of positional embeddings is missing. Therefore they are not used when mapping the patches into a latent embedding space l using a linear projection.
We also show the overview of the transformer encoder architecture in Figure 2, to help with the explanation stated below. The input of the first Transformer layer z 0 is calculated as follow:
where z 0 is mapped into a latent n-dimensional embedding space using a trainable linear projection layer and E is the patch embedding projection. These patches are then fed into the vision transformer specifically the VIT-B16 (Dosovitskiy et al., 2020a) variant. There are L Transformer layers which consist of multi-headed self-attention (MSA) and multi-layer perceptron (MLP) blocks. At each transformer layer ℓ, the input of the self-attention block is a triplet of Q (query), K (key), and V (value), which are computed from the output of the previous layer by matrix multiplication with learnable parameters of weight matrices. The self-attention in the attention head AH is calculated as:
where d is the dimension of self-attention block. MSA means the attention head will be calculated m times by independent weight matrices. The final MSA(z ℓ-1 ) is defined as:
The output of MSA is then transformed by an MLP block with residual skip connection as the layer output as:
where Norm means the layer normalization operator. Finally we use a fully connected layer to decode the (Moon et al., 2019) 108.1 HDNET (Lin and Lee, 2020) 69.9 Ours 68 output of the vision transformer.
Setting Details: The TransFocal model is trained with images of varied resolutions. The learning rate used is 1e-4. The weight decay is set to 1e-2. The batch size is set to 4. We adopt the Adam optimizer (Kingma and Ba, 2014) for training. We train the model until performance plateaus on validation set.
Training and Evaluation Datasets: A dataset is created using the Pano360 dataset (Kocabas et al., 2021c). Pano360 consists of real panoramas taken from flickr (Flickr, 2022) and synthetic panoramas. Due to a portion of flickr images being inaccessible, we were unable to recreate the complete dataset. The dataset is split for training and evaluation purposes. The code for creating the dataset is available at (Kocabas et al., 2021b). Note that as the image generation parameters are randomized, it is difficult to recreate exact the same dataset.
Here we report some performance comparison on multiple datasets and a ablation study in using the Absolute-ROMP. (Kocabas et al., 2019) 82.9 51.9 ROMP(HRNET-32) (Sun et al., 2020) 76.7 47.3 Absolute-ROMP(HRNET-32) 84 50.5
The root joint localization results on Human3.6M dataset are shown in Table 1. The baselines reported in the first 3 rows follow a two-stage approach (Moon et al., 2019), where 2D pose and 3D pose are estimated separately, and an optimization process is adopted to obtain the global root joint location that minimizes the re-projection error. The baseline "w/o limb joints" refers to optimization using only head and body trunk joints. The baseline "with RANSAC" refers to randomly sampling the set of joints used for optimization with RANSAC. The baseline results are taken from the figures reported in (Moon et al., 2019). We also compare with the state-of-the-art approaches (Moon et al., 2019;Lin and Lee, 2020). In (Moon et al., 2019) a multi-stage approach is used while in (Lin and Lee, 2020) a graph convolution network model is used. Our model is able to outperform the SOTA while maintaining inference in real time. Our system runs at ∼ 23 fps on a Nvidia Tesla V100 GPU. Our root joint localization head achieves a 68 mm accuracy. We also showcase the performance of Absolute-ROMP on the MUPOTS dataset in Table 2. Our model has comparable performance to the SOTA. We also compare MJPJPE and PMPJPE performance in Table 3 which does not make use of the absolute depth as joints are evaluated after root alignment. Even with added depth map prediction, we are able to maintain comparable performance with the SOTA. We also show qualitative comparison with ROMP in Figure 3 which highlights the importance of absolute global coordinates. Thanks to absolute depth information while positioning the meshes, we improve the location accuracy therefore correctly placing people hugging each other.
Finally, in Table 4 we show case TransFocal performance against a SOTA approach CamCalib (Kocabas et al., 2021c). Our model is trained on a dataset created from partially available images of the Pano360 dataset, while we compare the performance with a model provided by the author (Kocabas et al., 2021c) pretrained on dataset created from complete Pano360 dataset. We evaluate on a portion of the dataset unseen by our model. As can be seen our model is able to outperform CamCalib by up to 40%.
We show the improvement in performance when we use a combination loss instead of just employing the Softargmax-biased-L2 loss when training TransFocal. We report mean error after training for 1 epoch while using Softargmax-biased-L2 loss alone and with cross entropy loss in Table 5. The cross entropy loss acts as a guide for the gradient descent direction when the model is starting out thus adding to the speed of convergence of the model .
We build upon an end to end one-stage network ROMP for monocular multi-person 3D mesh regression, by adding in absolute distance prediction to create our Absolute-ROMP. In order to eliminate the need for intrinsic parameters of the camera, we also design and train a focal length prediction network called TransFocal, which is a CNN+Transformer hybrid model. We evaluate our models on industry standard benchmarks. Absolute-ROMP shows competitive performance while maintaining real time performance, while TransFocal is able to outperform the state-of-the-art.
Future work could incorporate the absolute camera parameters eliminating the need for predicting the depth separately. This would require the use of accurate absolute multi-person 3D datasets in a variety of scenarios, such as the synthetic dataset AGORA (Patel et al., 2021). In addition, real-time clothes and texture prediction would be especially beneficial for VR and AR applications.
\section*{ACKNOWLEDGEMENTS}