In recent years, Human Pose Estimation has emerged as an important problem in computer vision, with numerous applications in fields such as surveillance [13], motion analysis [31], virtual and augmented reality [17,23]. Classical pose estimation algorithms relied on handcrafted features [9,12,25], but recent advances in deep learning have led to significant improvements based on learned features [33,41]. For instance, Convolutional Neural Networks (CNNs) have proven to be successful by exploiting spatial correlations among pixels.
The recent emergence of Vision Transformers (ViTs) has challenged the dominance of CNNs. Unlike CNNs, ViTs rely on self-attention mechanisms to model the long-range dependencies between patches, which has been shown to be highly effective [7]. Nevertheless, the computational complexity of ViTs presents a significant challenge for processing high-resolution images. In particular, the computational cost of ViTs scales quadratically with the number of input tokens, making them intractable for practical use.
To address this issue, several recent works have proposed various methods for reducing the number of tokens that need to be processed by ViTs, thereby lowering their computational cost. Token Learner [28] is one approach that aims to merge and reduce the input tokens into a small set of important learned tokens. Token Pooling [19] clusters the tokens and downsamples them, whereas DynamicViT [26] introduces a token scoring network to identify and remove redundant tokens. Although these techniques successfully reduce the computational complexity of ViTs in classification tasks, the additional pooling and scoring network can introduce extra computational overhead. Besides, the extension of these approaches to dense prediction tasks, such as human pose estimation, remains an open question.
In this work, we propose to reduce the computational complexity of transformer based human pose estimation networks by selecting and processing a small subset of patches that are most likely to contain body joints or limbs. The rationale behind this choice is that results from [40] suggest that pixels neighboring a joint or a limb are more informative about joint locations. We first propose two patch selection methods that utilize fast yet imprecise out-ofthe-shelf pose estimation networks to guide the selection process. We then propose a patch selection method that uses learnable joint tokens to progressively select the most informative patches. As a result, our approach significantly enhances computational efficiency, albeit with a minor trade-off in accuracy. To validate the effectiveness of our proposed methods, we conducted experiments on six 2D human pose estimation benchmarks. The experiments show that our approach significantly reduces computational complexity (30% to 44% drop in GFLOPs), while only having a minimal drop in accuracy (between 0% and 3.5%).
2 Related Work
Human Pose Estimation is an essential task in computer vision that involves identifying and estimating the location of human body joints from 2D images or videos. In recent years, deep learning methods have been successful in 2D human pose estimation, with most meth-ods employing CNNs to learn a mapping between the input image and the corresponding 2D pose. There are two main deep learning pipelines used in single person pose estimation: regression-based and heatmap-based approaches. Regression-based methods directly map the input image to 2D joint positions [33], while heatmap-based methods predict approximate joint locations using 2D Gaussian heatmaps centered at the body joint [36]. The heatmap-based approach is effective and commonly used, and will be used for inference in this work although both will be utilized for training.
Multi-person pose estimation is a difficult task that requires determining the number of people and their positions, as well as grouping key points. Two main approaches are top-down and bottom-up. Top-down approaches [21,32,38,40,42] use person detectors to extract boxes from input images and then apply single-person pose estimators to produce multiperson poses. In contrast, bottom-up techniques [3,4,22,30] identify all body joints in a single image and group them for each person. This work will follow the top-down approach, which has been shown to be effective.
The Vision Transformer (ViT) architecture proposed by Dosovitskiy et al. [7] has demonstrated remarkable performance on image classification tasks. Consequently, several transformer based architectures have been proposed for human pose estimation. Transpose [40] is a transformer network that estimates 2D pose using a CNN-based backbone. Token-Pose [15] is another transformer based on explicit token representation for each body joint. HRFormer [42] is a transformer that adopts HRNet design along with convolution and localwindow self-attention. ViTPose [39] is a ViT-based approach that uses a shared encoder training on multiple datasets to improve performance. These architectures demonstrate the effectiveness of transformer-based models in human pose estimation.
However, vision transformers have quadratic computational complexity. To improve the efficiency of ViT, researchers have proposed methods such as sparsifying the attention matrix [5,27], token pooling [19], and estimating the significance of tokens [26]. Hierarchical Visual Transformer [24] removes redundant tokens via token pooling, while Token-Learner [28] introduces a learnable tokenization module. Adaptive Token Sampler [8] adaptively down-samples input tokens, which assigns significance scores to every token based on the attention weights of the class token in ViT. Similarly, EViT [16] determines tokens' importance scores via attention weights. However, it is still an open question whether these approaches can be extended to dense prediction tasks, such as human pose estimation. In our third patch selection method, we will follow a similar approach to [8,16] and extend it to the human pose estimation task.
In this section, we describe our approach to human pose estimation, which includes revisiting the standard ViT-based method, incorporating learnable joint tokens and patch selection techniques. We present the overall architecture of our method in Figure 1a. Given an input image X 2 R H⇥W ⇥3 , the task is to find a mapping from X to Y 2 R J⇥2 , where J is the number of body joints, for each person in the image. We first embed X into patches of size 16 ⇥ 16, resulting in a set of patch tokens P 2 R N⇥C , where N = H 16 ⇥ W 16 and C represents the channel dimension. We then extend the patch tokens with J learnable joint tokens, J 2 R J⇥C , that explicitly embed each one of the joints that are later used to regress the joint 2D positions in the image, resulting in a concatenated set of input tokens I 2 R (N+J)⇥C . The patch and joint tokens are then fed to a standard ViT encoder with L transformer blocks, each of which consists of a multi-head self-attention (MSA) layer and a feed-forward network (FFN).
Here, we revisit the standard self-attention mechanism. Later, we will show how this mechanism is modified in the case of the joint-token-based patch selection method. In the selfattention mechanism, the output tokens O and the attention matrix A are computed as
where
⇥C are, respectively, the queries, keys, and values, which are computed from the input tokens I as in the standard ViT [7].
Given the final feature map of the patches produced by ViT, denoted as
16 ⇥C , we use a classical decoder with two deconvolution blocks, each with a deconvolution layer, batch normalization, and ReLU activation to estimate the heatmap of J joints. Similarly, a LayerNorm layer followed by a fully connected MLP layer is used to directly regress the J joint coordinates from the joint tokens. In this way, the joint tokens are enforced to learn the important joint-level information to be able to successfully regress the joint 2D positions.
Although ViT can model long-range dependencies and is able to generate a global representation of the overall image, the computational complexity increases quadratically with respect to the number of tokens. However, not all patches in an image contribute equally to the human pose estimation task. Recent research [40] indicates that the long-range dependencies between predicted joints are mostly restricted to the body part regions. Therefore, computing MSA between every patch in the image while only a few patches are relevant to the body parts is unnecessary. To this end, we propose three methods to select a small number of relevant patches while discarding irrelevant and background patches without retraining the vision transformer. By selecting only the relevant patches, we can significantly reduce the computational complexity as shown in [8,20,26] for the classification task.
The first two approaches utilize an off-the-shelf lightweight pose estimator to guide the patch selection. Our first approach is based on a breadth-first neighboring search algorithm that selects body joint patches and its neighbors given estimated pose predictions, as outlined in Section 3.2. In our second approach, we extend the first approach to select patches formed by a skeleton of the joints. Here, the objective is to select body patches where the lines formed by body joint pairs cross. To accomplish this, we utilize Bresenham's algorithm to select the relevant patches, as outlined in Section 3.3. It is important to note, however, that by selecting a few patches of the image and processing them with the ViT encoder, we only get the features of the patches that were chosen. However, we must create a feature map for all patches for further processing. As a result, since the goal of the task is to produce a Gaussian heatmap centered at the body joint and zero elsewhere, we fill the non-body-part patches with zeros. Whilst the aforementioned methods can remove irrelevant patches before they are processed with ViT and thus enhance its efficiency, their reliance on the accuracy of offthe-shelf pose estimators is a limitation. As a result, we present an alternative approach for for j 1 to J do 5:
. get column 6:
. get row 7:
for (k, l) 2 N (x j , y j , n) do 8:
x n = (x j + k)
y n = (y j + l)
10: p = y n ⇥ c + x n 11: BP p . add patch to set 12: end for 13: end for 14: end function Algorithm 2 (Select body part patches formed by a skeleton ) Require: joint prediction B 2 R J⇥2 , body joint pairs P, patch size P 1: function SELECTSKELETONPATCHES 2:
for j 1 to J do
4: x j 0 = b B j x P c, y j 0 = b B j y P c 5:
for l 2 P( j) do 6:
x end for 20: end function automatically selecting body part patches via learnable joint tokens that enable the selection of relevant patches using their corresponding attention maps, as outlined in Section 3.4.
In our first approach, we assume we are provided with an estimate of the joint locations B 2 R J⇥2 via a lightweight pose estimation network, where J denotes the number of joints. To select the body joint patch and its n neighboring patches, we employ a Breadth-First Search (BFS) algorithm, as presented in Algorithm 1. Specifically, for every joint located at 2D patch location (x, y) 2 B, we identify the nearest four neighboring patches located at (x, y + 1), (x, y 1), (x 1, y), and (x + 1, y), and store them in a queue. We continue searching for neighboring patches until we have selected n patches, ensuring that we do not revisit any previously visited patches. To provide a visual representation of this approach, we depict in Figure 1b a skeletonized human figure where the red patches represent the joint patches and the set of orange patches correspond to the selected neighboring patches.
A limitation of the aforementioned neighboring selection method is that it only covers the body joint patches and its neighbors. Our second approach extends this method to encompass all patches between joints. This can lead to improved performance with a minimal increment in the computational complexity. To achieve this, we adopt a different strategy by identifying the patches where the line formed by the body joint pair (segments) crosses, based on Bresenham's algorithm [2]. Originally proposed as a canonical line-drawing algorithm for pixellated grids, our extension of Bresenham determines the patches that need to be selected in the line between (x 0 , y 0 ) and (x 1 , y 1 ), corresponding to the start and end 2D locations of patches containing body joints. As we move across the x or y axis in unit intervals, we select the x or y value between the current and next value that is closer to the line formed by the body joint pairs. To make this decision, we require a parameter e. Our objective is to track the slope error from the last increment, and if the error exceeds a certain threshold, we increment our coordinate values and subtract from the error to re-adjust it to represent the distance from the top of the new patch, as presented in Algorithm 2. As depicted in Figure 1b, the set of blue patches represents the patches selected by this approach.
The limitation of the first two patch selection methods is that they rely on the performance of the lightweight off-the-shelf pose estimator. This limitation can become especially problematic when dealing with complex scenes, as the accuracy of the pose estimator is often compromised by occlusion, motion, or variations in camera perspective. As a consequence, this might result in the selection of irrelevant patches and removing important patches, leading to suboptimal performance. Therefore, a more robust approach is required that can adapt to these challenging scenes without the need for an off-the-shelf pose estimator.
Selecting most informative patches via learnable joint tokens. We propose to overcome this limitation by selecting patches using the learnable joint tokens, which serve as a powerful feature representation for distinguishing the relevant body part patches. Specifically, we aim to determine the importance of each patch in relation to the joint tokens, thereby enabling us to select the most informative body part patches. To achieve this, we harness the attention matrix similar to [8,10,16], as the values in A serve as the weight of contribution of input tokens to output tokens. For example, A N+1,1:N denotes the contribution weights of the N patch tokens to the (N + 1) th output token, i.e. first joint token. Thus, we can calculate the average contribution weight of a patch token l to the J joint tokens, as follows:
Following [8], we take the norm of V l into account for calculating the importance score. Thus, the importance score of the patch token l is:
where l, k 2 {1,. .., N}. Once the importance scores of each patch token have been computed, we select L patch tokens with the highest scores for further processing, where L ⌧ N.
Pruning attention matrix. Our subsequent step involves pruning the attention matrix A 2 R (N+J)⇥(N+J) by selecting the rows that correspond to the chosen L patch tokens and J joint tokens, designated as A s 2 R (L+J)⇥(L+J) . We then compute the output tokens O s 2 R (L+J)⇥C , given by:
These output tokens are then passed as input for the next blocks.
Refining background patches via joint tokens. Although only O s will be processed in the next blocks of ViT, the non-selected patch tokens will still be used during the heatmap decoding. Therefore, it is important to have a refined representation of the non-selected patch tokens before they are excluded from further processing in the next blocks. Thus, we propose an efficient method that updates these tokens using the joint tokens. This approach is motivated by the fact that joint tokens learn global information and therefore can be used to update the patch tokens in a computationally efficient manner without the need to compute contributions from all tokens, which can be computationally expensive. We start by selecting the rows of the attention matrix A that correspond to the non-selected patch tokens and the columns that correspond to the joint tokens, resulting in a sub-matrix A o 2 R (N L)⇥J . We then update the non-selected patch tokens using the joint tokens as follows:
where V j corresponds to the values of the joint tokens.
In our experiments, we employ the common top-down setting for human pose estimation. We follow most of the default training and evaluation settings of the mmpose [6] framework but use the AdamW optimizer with a learning rate of 5e 4 and UDP as a post-processing method. We use ViT-B and ViT-L as backbones and refer to the corresponding models as EViTPose-B and EViTPose-L. The backbones are pre-trained with MAE [11] weights and trained with multiple datasets as in [39].
The proposed methods are evaluated on six 2D pose estimation benchmarks: namely MPII [1], COCO [18], AI Challenger [37], CrowdPose [14], JRDB-Pose [35], and OCHuman [43].
The datasets are challenging and diverse, with varying numbers of images, person instances, and annotated joints. The first five datasets are used to train and test the proposed method, and OCHuman is only used to test the models in dealing with occluded scenes.
On the MPII benchmark, we adopt the standard PCKh metric as our performance evaluation metric. PCKh [1] is an accuracy metric that measures if the predicted joint and the true joint are within a certain distance threshold (50% of the head segment length). On the remaining benchmarks, we adopt standard average precision (AP) as our main performance evaluation metric. AP is calculated using Object Keypoint Similarity (OKS), which measures how close the predicted joint location is to the ground-truth joint. Additionally, we will evaluate the computational complexity of the models by measuring the total number of Floating Point Operations (FLOPs) that each model needs to perform.
The performance of our proposed methods and other convolutional and transformer-based methods on three datasets, namely the COCO val set, the MPII test set, and the OCHuman test set, are presented in Table 1. We used LiteHRNet [41], a lightweight and less accurate pose estimation network, to guide the first two patch selection methods. Additional results of the joint-token-based patch selection method on AI Challenger val set, CrowdPose test set, and JRDB-Pose val set is presented in Table 2. The remaining patch selection methods were not evaluated on these benchmarks as we were not able to find lightweight pose estimators trained on them. Our experiments show that EViTPose without patch selection outperforms all, but is computationally expensive. However, our proposed patch selection methods proves to be beneficial in this regard, as it significantly reduces computational costs while maintaining high accuracy. For instance, we achieve a reduction of 30% to 44% in GFLOPs with a slight drop in accuracy ranging from 1.1% to 3.5% for COCO, 0% to 0.6% for MPII, and 0.7% to 3.5% for OCHuman. We can also control the drop in accuracy by changing the number of patches to be selected. The trade-off between performance and computational complexity for the neighboring and joint-token-based patch selection methods is depicted in Figure 2. The neighboring and skeleton patch selection remove irrelevant patches before they are processed by ViT, while the joint-token-based selection method learns to remove them on the fly. Thus, the first two approaches prioritize efficiency over accuracy by removing patches early on. For example, they are effective for addressing the low end range in Figure 2, where the joint-token-based selection method performs poorly. Some qualitative results of our method on sample images from the benchmarks are illustrated in Figure 4. Ablation. We conducted a run-time comparison (measured in frames per second, FPS) among EViTPose, ViTPose and TokenPose, presented in Table 3. The results show that our Joint-Token-based Patch Selection method (EViTPose-B/JT) achieves an 88% reduction in GFLOPs and 10⇥ increase in FPS with respect to ViTPose-H, with a minimal drop in accuracy of upto 2.9%. Furthermore, we evaluate the effect of the lightweight pose network in the performance of EViTPose with the first two patch selection methods as shown in Table 3. EViTPose maintains consistent performance across three different lightweight pose estimation models, namely LiteHRNet [41], MobileNetv2 [29,34], and ShuffleNet [34,44].
In this work, we have proposed EViTPose, a Vision Transformer-based human pose estimation network with patch selection methods that greatly reduces the computational com- plexity of ViTs while maintaining high accuracy. The proposed methods involve selecting and processing a small subset of patches that contain the most important information about body joints. Two of the methods utilize a fast yet imprecise out-of-the-shelf pose estimation network to guide the patch selection process, while the third method uses learnable joint tokens to progressively select the most informative patches. The experimental results on six widely-used 2D pose estimation benchmarks demonstrate that our methods significantly improve speed and reduce computational complexity, with reductions ranging from 30% to 44% in GFLOPs, with only a slight drop in accuracy, between 0% and 3.5%. Acknowledgments. The authors thank Carolina Pacheco and Yutao Tang for their valuable input and feedback throughout the development of this work. This research is based upon
© 2023. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.