Hands are crucial in many industrial computer vision applications, such as augmented reality, virtual reality, or human-computer interaction. Recognizing hands with vision systems is necessary to interact between people and digital devices. Therefore, understanding hands with computer vision systems has been deeply explored through hand tracking [41,54], hand pose estimation [18,21,22,27,29], grasp detection [10,47], hand gesture recognition [58], multi-view prediction [29], and hand-action classification [50]. These works require segmenting hands from the background to increase the accuracy of performance. Most of these applications require first-person-view images of hands in actions with tools. However, segmenting hands interacting with tools from the background is a challenging problem because (a) fingertips are heavily occluded by the hand dorsum and tools in the first-person view, (b) tools are held with various grasps, and (c) shapes of tools or objects are infinite. The traditional approach to create a RGB hand segmentation video dataset is through manual pixel-wise labeling [5,7,32,50]. However, time and cost of person-inthe-loop segmentation grows linearly as the number of frames increases, which reduces the scalability. Therefore, developing an efficient segmentation method is important for creating a segmented hand video dataset.
We utilize hand temperature as prior information for the labeling process. However, pixels from Long-Wave InfraRed (LWIR) thermal images of hands may falsely include pixels from the surroundings that share the same temperature. In our method, we utilize crowd workers to provide an Axis-Aligned Bounding Box (AABB) to localize the detailed boundary of each hand. We further relax the AABB creation task by training a tracker with a small amount of AABB results. The tracker learns the shape features of hands and then uses them to estimate an Oriented Minimum Bounding Box (OMBB) for each hand. Therefore, we use both spatial and thermal features of hands to segment them from the background and tools. Our approach is effective regardless of finger tips or finger joints occlusion and does not require hand pose ground truth. We prove that our method is much more efficient than the traditional pixel-wise labeling tasks (sec. 6) while maintaining a high performance (sec. 7).
Optimizing deep neural networks with a single modality may lead to failures when the network fails to extract distinctive features from the input source. Multiple modalities are used to provide distinctive features to the networks [14,56,60] and has been an emerging area [11,26,44] due to the enhancement of computation power and sensors. Human body temperature is relatively constant [9] and has been widely used in pedestrian detection [28], biological image processing [19], and gesture recognition [4]. An additional advantage of LWIR is that it is invariant to colors, textures, and lighting conditions. Color information may mislead vision systems distinguishing shape features. Therefore, we create a multimodal (LWIR, RGB, and depth) hand segmentation video dataset which consists of 790 sequences and 401,765 frames of "hands using tools" videos. Compared to the other existing hand segmentation datasets, our dataset contains three different modalities and headmounted camera Inertial Measurement Unit (IMU) information. Sample frames from our dataset are shown in Figure 1 and a detailed comparison with other hands datasets is shown in Table 1.
With the video dataset, we analyze fusing three modalities with DeepLabV3+ [13] and benchmark five different state-of-the-art segmentation methods [13,49,50,51,61]. We observe that the neural networks can automatically learn important cues from three different modalities: LWIR, RGB, and depth. The jointly-learned features of these three modalities prevent confusion between hands and backgrounds.
The main contributions of this paper are as follows:
• We collect large-scale action videos in a first-person view which contain LWIR, RGB, depth, and IMU information. This dataset can be used for hand segmentation research using multiple modalities.
• We develop a framework that can significantly reduce segmentation efforts by leveraging hand temperature for creating pixel-wise hand segmentation ground truth when a person is holding tools. Our method does not require hand pose labels nor a hand mesh model.
• We analyze the effectiveness of multiple modalities for hand segmentation task with deep neural networks and found the optimal combination which is fusing thermal (LWIR), RGB, and depth modalities.
Efficient annotation methods are crucial in creating labels for large-scale video datasets.
Although complex boundaries can be traced manually, labeling image/video datasets with object masks is extremely time consuming [34]. A successful method is to have a neural network produce polygon annotations of objects interactively using humans-in-the-loop to improve the accuracy [1,31,39]. The machine can provide the human with information to manipulate [3] and generate segments [2], matting layers [59], or boundary fragments [43].
Sequences in video datasets consist of similar frames which contain redundant information.
For moving objects especially, several notable and established annotation tools with autotracking demonstrate great performance on efficiency improvement [6,46]. Unlike these works, we extract shape features from thermal frames for pixel-wise hand segmentation. Therefore, our annotation pipeline is invariant to colors and textures. The invariance property improves the quality of labels and efficiency.
Pixel-wise hand segmentation dataset with deep neural networks has been widely used for segmenting objects in videos [12,40,42,61]. Deep-learning-based methods use convolutional neural networks to predict each pixel in an image by extracting feature representations. One of the popular structures is an encoder-decoder structure which projects a high-dimensional image into the latent vector and decodes the latent vector into class-wise pixel space [13,35]. Researchers create hand segmentation datasets by manually drawing polygon or coloring the hand on RGB frames [33,38,48,48,50]. Other works use hardware sensors to predict joint position and render with mesh models; however, the sensors are visible in the RGB images [7,57]. Alternatively, studies showed good performance when utilizing hand pose estimation and mesh models to segment hands in frames [45,62]. Our method doesn't require training a hand pose network [20] to generate high accuracy on hand pose estimation under heavily occluded situations.
Multi-modal data is widely used to capture robust features because of color and texture invariance and its expression of an object in an explicit 3D information [15,17,37,52]. Therefore, LWIR is widely used in detecting objects [52] and controlling unmanned aerial vehicles due to its outstanding ability of identifying objects with a distinct temperature from the surroundings [24,49,55]. MFNet [23] showed that fusing LWIR and RGB frames significantly enhances the segmentation performance. Luo et al. [36] uses thermal information to segment coarse point clouds scenes. Thus, many RGB pixels can not be labeled from the coarse labeled point clouds. Most works with LWIR sensors are for scene segmentation and autonomous driving system. Unlike other works, we use LWIR sensors to threshold the hand temperature and remove backgrounds omitting a similar wavelength of thermal radiation with spatial features for hand segmentation dataset creation (sec. 8).
We record our videos with a low-cost non-radiometric thermal camera, Flir Boson 320, to capture relative LWIR data. RGB resolution then is rescaled to match resolution of the LWIR sensor with two camera frustums. We use an Intel D435i depth camera for RGB, depth, and IMU information acquisition. These sensors are placed within a 3D printed case and mounted in front of a helmet to make the camera location consistent over sequences.
To segment hands with LWIR frames, we narrow down the search space by finding a Thermal Mask (TM), denoted I tm , with LWIR frames, denoted I raw lwir , and I lwir = T (I raw lwir ) where T is transformation function that transforms a LWIR plane to a RGB plane.
The bounded value for a pixel in the target frame corresponding to a spatial location (i, j) in I tm is defined as follows:
,where a and b are upper bound and lower bound of hand temperature. I tm ∈ [0, 1] H×W and I lwir ∈ R H×W . We map the I raw lwir onto the RGB frame, denoted I rgb , with depth maps. To align depth maps and I raw lwir , we find the spatial relationship between the I raw lwir and depth camera. A projection of an object in pixel space is derived by multiplying the camera matrix (K T for LWIR camera and K D for depth camera) with an object point.
,where
T , a projected point in depth and LWIR camera pixel plane, respectively. P D and P T are an object point in depth and LWIR camera coordinate, respectively. λ is a scale factor and w D is depth value in camera space. The spatial relation between two cameras is defined by equation 6 where R is a 3D rotation matrix and T is a translation matrix.
By combining equation 4, 5, and 6, we can get an equation:
By solving equation 7, we transform I raw lwir to the depth plane and depth plane to the RGB plane. The detail of solving equation 7 is explained in the supplementary document. The RGB and depth frames are aligned using the Intel RealSense API. the different resolution and field of view between two cameras are adjusted using intrinsic and extrinsic camera parameters. After the alignment, we threshold the hand temperature by setting the lower and upper bounds in human temperature as depicted in Figure 2 b. These bounds are manually captured for every sequence and used as priors that segment hands from surrounding backgrounds and the hand-held objects. To create accurate bounds, we overlapped the I raw lwir on the depth maps as seen in Figure 2 column a. Finally, we get the segmented hands by filtering thermal mask (see Figure 2 column d). Removing mislabeled pixels and identifying orientation We occasionally observe mislabeled pixels from backgrounds that have similar temperature as hands. To remove these mislabeled pixels, we use a tracking algorithm, named SiamMask [53], to localize hands with OMBBs shown in Figure 3. We train the tracker with I tm and AABBs of the hand. We use Amazon Mechanical Turk (AMT) to crowdsource the creation of hand AABBs. For training the tracker, I tm and corresponding AABBs are used as targets, and I lwir is used as inputs. Therefore, the tracker is color and texture invariant, which improves tracking performance of the tracker (sec. 7). After the training process, given initial AABBs, the tracker predicts OMBBs and classifies them sequentially OMBBs as a left hand or a right hand through the frames. This implies that the tracker learns hand shape features. These OMBBs are used to remove mislabeled pixels by intersecting I tm and OMBBs.
In this section, we show the efficiency of our proposed segmentation method, the performance of the tracker, and the analysis of multi-modal sources for hand segmentation, and we also benchmark our dataset with five different segmentation methods. We consider the hand segmentation problem as a two-class segmentation task, and we plot the maximum probability between the two classes, background and hands, per pixel for the prediction mask creation. For evaluation metrics, we use the Intersection over Union (IoU) of the hand (hIoU) and the background (bIoU). We define the mean IoU (mIoU) as the mean of these two class IoUs. For the test dataset, we use manually-annotated labels which consist of sequences that are used in neither the training set nor the validation set.
To segment hands from objects, we create a pixel-wise hand segmentation dataset with subjects holding objects and tools. The dataset consists of 401,765 frames and 790 sequences. Our dataset has a large number of sequences and frames compared with the other datasets as shown in Table 1. We manually annotated 13,792 frames from 136 sequences to create a test dataset. The video dataset contains five subjects, 15 actions, and 23 tools. The distribution of the dataset across the actions and tools is plotted in the supplementary document. For annotating per pixel label of hands, we use I lwir as prior knowledge and used a tracker to identify orientation of hands as well as whether a hand is a left hand or a right hand.
We evaluate the accuracy of I tm with manually-labeled frames. I tm is defined by the temperature of the hands. It shows fairly reasonable accuracy of 0.849 in hIoU. We find that I tm reduces false positive in the sequences where non-hand area has the same temperature as hands. These mislabeled pixels are the main reason of the hIoU degradation. We remove these mislabeled pixels with the tracker given initial AABBs [53]. Utilizing I lwir improves the efficiency of the manually labeling frames. We profile the amount of time that it takes to annotate frames using four different methods for pixel-wise segmentation labeling. First, annotators use PolyRNN++ [1] with I rgb . Second, annotators label hands on a tablet with a tablet pen given I rgb . Third, annotators label hands on a tablet with a tablet pen given masked I rgb = I rgb I tm , where is the Hadamard product. Lastly, annotators draw AABBs on top of I rgb . Ten people annotated a random sample of twenty frames with the four different methods. We then averaged the annotation time, yielding the results in Table 2. We find that drawing AABBs is 24 times faster than the other methods. This implies that our method is 24 times faster than PolyRNN++ since our method intersects AABBs on I tm . Additionally, the third method is two times faster than the second approach and six times faster than using PolyRNN++. Masking hands with I tm significantly narrows down the region for annotators, which reduces the labeling time. To validate the quality of the annotation methods, we randomly sample 136 sequences and manually annotate 13,792 frames. With these manually annotated frames, we evaluate the IoUs of I tm with and without AABBs labels (see Table 3). The total cost of our method is $2,343 for annotating 401,765 frames, with $8/hr as the minimum wage of our contractor. To annotate 401,765 frames with a tablet and pen, 76 seconds per frame, it requires $67,853, which is 28.96 times more expensive than our method. The time cost of PollyRNN++ is 122 seconds per frame, resulting in $108,923 for the pixel-wise annotation, which is 46.49 times more expensive than our method.
Avg. Time (second) Drawing polygon with PolyRNN++ [1] 122 Painting hands on I rgb with a tablet pen 76 Painting hands on I rgb with a tablet pen 24 Drawing AABBs on I rgb 5
Table 2: Comparison table of annotation average time cost per frame. I rgb = I rgb I tm . Lower average time is better.
We use SiamMask as our tracker [53] and train the tracker with a seeding dataset consisting of 518 sequences that have 11,718 frames labeled by crowd workers. The dataset is divided into 441 sequences that have 7,882 frames for training and 77 sequences that have 3,836 frames for validation. The tracker is trained in two ways: with I rgb frames and with I lwir frames. For evaluation, we use the manually labeled frames and metrics from Section 4.2.
From the evaluation, we find that the tracker with I lwir frames outperforms the others as shown in Table 3. The tracker with I rgb tends to detect more forearm than the tracker with I lwir as shown in Figure 4. This implies that the tracker with I lwir is more sensitive in finding convex shape of wrist than the tracker with I rgb , yielding better orientation of the hand and tighter OMBBs. The tracker performs well in most of cases; however, we need to re-initialize it with AABBs when the tracker fails to estimate the next frame. We also find that the tracker fails to track the hands when the two hands are heavily overlapping. In this case, we need to manually draw the OMBBs. Conventionally, creating pixel-wise segmentation is done by manually drawing polygons. The major drawback of this method is that polygon can not define smooth curves, not like our method. Particularly in hand-with-object cases, the boundary of fingers and objects creates many holes and curves. Our method can label pixel by pixel and represents smooth curves. Therefore, mIoU of PolyRNN++ is 0.895 [1] which is not as accurate as our method which is 0.923.
We analyze the effect of multi-modal sequences, {I rgb ,I lwir ,I depth }, for hand segmentation by conducting seven ablation studies using all possible combinations of {I rgb ,I lwir ,I depth } as input modalities. We perform the seven ablation studies to find out how I lwir contributes in training neural networks. For all experiments in this section, we use randomly sampled 50K frames and split into two sets as the following: 40K frames as train dataset and 10K frames as test dataset. The frames in the test dataset are labeled manually. We use DeepLabV3+ [13] as base model and add additional encoders to fuse additional modalities. Our fusing method is detailed on the supplementary document. The rationale of using DeepLabV3+ is that it outperforms other methods [50,51,61] in hand segmentation benchmark experiments, only using RGB, as shown in Table 4. It also has the fewest parameters. We use ResNet 101 [25] which is pre-trained on the ImageNet [16] as a backbone network. All experiments use an equal number of encoders as the number of input modalities. From experiments, we found that I lwir guides the network in finding better minima by observing both the loss drops and performance improvement as shown in Figure 5 when the I lwir is used. Including I lwir enhances the performance of hand segmentation by 5% in hIOU score compared to {I rgb , I depth }. We also find that including IMU information improves 0.006 mIoU improvement. Increments are observed for bIOU and mIOU scores as well. Therefore, I lwir is a robust feature for hand segmentation. The three modalities contain complementary properties, which generates robust features, compensates weak points of each other, and leverages their advantages. The models have been trained using Stochastic Gradient Descent (SGD) [8] and the ADAM optimizer [30] with initial parameters: learning rate as 0.001, β 1 = 0.9, β 2 = 0.999. We decay the learning rate by 0.1 every 250K steps. Additional configurations of the experiments are listed in the supplementary document. We use a single TITAN RTX GPU and an Intel i7-6850K CPU for the experiments.
To validate the performance of using multiple modalities, we compare our method with five state-of-the-art segmentation methods [13,13,50,51,61] which use I rgb and segmentation networks [49] and jointly use I rgb and I lwir as input modalities. We use the same dataset as Section 4.4 and hyper-parameters listed on the original papers. We notice that RTFNet [49] performs second-best among all methods, indicating I lwir provides the most meaningful prior knowledge for segmenting hands in frames. DeepLabV3+ * with three modalities outperforms the second-best method, RTFnet, by 4% in hIoU and 30% fewer parameters, as shown in Table 4.
In this work, we propose a robust and efficient pixel-wise hand segmentation method and a multi-modal dataset. We record rich sequences with three different image modalities and IMU information of first-person-view images with pixel-wise hands and action labels. We found that using multiple modalities achieves 4% better hIoU when compared to the existing