The deep convolutional neural networks for computer vision tasks (e.g. classification [37,39], detection [26], segmentation [2], etc.) are highly relied on large-scale labeled datasets [19,40]. Collecting and annotating the large-scale datasets are usually expensive and time-consuming. To fa-cilitate 3D computer vision research, more and more 3D datasets such as mesh and point cloud data have been recently proposed. Compared to the annotation process of 2D image data, 3D point cloud data are especially harder to annotate and the cost is more expensive.
To learn features from unlabeled data, many self-/unsupervised learning methods are proposed for images, videos [11,18,24], and 3D point cloud data [13] by training deep neural networks to solve pretext tasks with automatically generated labels based on attributes of the data such as clustering images [3,33], playing image jigsaw [32], predicting geometric transformation of images or videos [9,17], image inpainting [35], reconstructing point cloud [57], etc. The learned features through these processes are then used as pre-trained models for other tasks to overcome over-fitting and speed up convergence especially when training data is limited.
Recently self-supervised feature learning on 3D point cloud data attract more attention including auto-encodersbased methods [1,8,57,59], generative model-based methods [28,48,54], and context-based pretext task method [13,58]. The auto-encoders-based and generative-based methods learn features by generating or reconstructing the point cloud data and have obtained very competitive performance on the 3D recognition benchmark [57]. However, by optimizing the loss for generation or reconstruction tasks, these networks suffer from modeling low-level features and compromising their ability to capture high-level features from the point cloud data.
In this paper, as shown in Fig. 1, we propose a novel idea to explore how to use the abundant relations of different views and modalities of 3D data (e.g. mesh, point cloud, rendered shading images, rendered depth images, etc.) as supervision signal to learn both 2D and 3D features without using any human annotated labels. The main contributions of this paper are summarized as follows:
• We design a new schema to jointly learn both 2D and 3D features through solving two parallel pre-defined pretext tasks: 1) Cross-modality task -to recognize whether two data in different modalities (3D point cloud and 2D image) belong to the same object; 2) Cross-view task -to minimize the distance of 2D image features in different views of the same object while maximizing the distance of 2D image features from different objects.
• The discriminative 2D and 3D features learned by the self-supervised schema are used as pre-trained models for other down-stream tasks such as classification, retrieval, and 3D part segmentation, etc.
• Extensive experiments on five different tasks (i.e. multi-view 2D shape recognition, 3D shape recognition, multi-view 2D shape retrieval, 3D shape retrieval, and 3D part-segmentation) demonstrate the effectiveness and generalization of the proposed framework. For the recognition tasks, our 2D and 3D models outperform the existing state-of-the-art unsupervised methods and achieve comparable performance as the supervised methods on the ModelNet40.
3D Point Cloud Understanding: Various methods have been proposed for point cloud data understanding and they can be categorized into three types: hand-crafted methods [5,20] which use hand-designed feature extractors to model the geometric features; deep neural networks on regular 3D data [4,7,12,22,31,38,46,47,49] in which the network usually operates on multi-view rendered images [46,47] or volumetric voxelized data [4,22,31,38,49]; and deep neural networks on unordered 3D data in which the network operates directly on the unordered point cloud data [27,29,37,39,51,53,55]. 3D point cloud data can be rendered into 2D images from different views to create multimodality data. 2D Unsupervised Feature Learning: Recently, many selfsupervised learning methods (also known as unsupervised learning) have been proposed to learn features from unlabeled data [3,9,11,17,18,24,32,33,35]. Usually, a pretext task is defined to train a network with automatically generated labels based on the attributes of the data. These methods fall into four groups: correspondence-based method (i.e. using the correspondence of two different modalities like visual and audio streams in videos as supervision signals) [25]; context-based methods (i.e. using context structure or similarity of the data as supervision signals) [3,9,32,33]; generation-based methods (i.e. using the learned features in the process of generating images or videos such as Generative Adversarial Networks and Autoencoder) [35]; and free semantic label-based methods (i.e. using the automatically generated labels by game engines or some traditional methods) [34]. The 2D self-supervised learning has been well studied recently, and some methods have been successfully adapted to the 3D self-supervised feature learning [13,42,58]. 3D Self-supervised Feature Learning: Several selfsupervised learning methods have been proposed to model features from unlabeled 3D point cloud data [5,8,20,28,48,54,57,59]. Most of these methods are auto-encoder based [1,8,57,59] to learn the features in the process of reconstructing the point cloud data or generative-based methods [28,48,50,54] to learn the features in the process of generating plausible point cloud data. Recently, a few work attempted to learn features by designing novel pretext tasks [13,42,58]. Sauder et al. proposed to learn features by recognizing the relative position of two segments from point cloud data [42]. Zhang et al. proposed EdgeConv to learn features by verifying whether two segments are from the same object and then boosting the performance of a cluster task [58]. Hassani et al. proposed a multi-task learning framework to learn features by optimizing three different tasks including clustering, prediction, and reconstruction [13]. However, all these methods only focus on learning one type of feature for 3D shape data while ignoring the inherent multi-modalities of different data representations. In this paper, we propose to learn two different types of features, 2D image features, and 3D point cloud features, by exploiting the correspondences of cross-modality and crossview attributes of 3D data.
Preparing 2D images in multiple views and 3D point cloud data from mesh objects is essential for our proposed self-supervised 2D and 3D feature learning. The details of the data generation, the architecture of the framework, and model parameterization are introduced in the following sections.
As shown in Fig. 1, two types of training sets are generated from 3D object datasets, i.e., multi-view rendered image set and sampled point cloud set, for learning 2D and 3D features. 3D objects are typically represented in polygon meshes as collections of vertices, edges, and faces, etc. See Section 3.3 for specific input samples for the framework.
Multi-view image generation: Following [46], the Phong reflection model [36] is employed as the rendering engine to generate rendered images in different views from 3D polygon meshes. By given a 3D polygon mesh m from a 3D object set M , a spherical coordinate system is defined with the centroid of m as the center for the system. The centroid for each m is calculated as the average of all mesh face geometric centers of m, while the mesh face centers are weighted by the corresponding mesh face areas. To project m to multi-view 2D planes, V virtual cameras (viewpoints) around m are randomly placed for each object along a sphere surface with radius R (see Fig. 1). Each virtual camera is arranged by an azimuthal angle (randomly selected from 10 to 340 degrees) and a polar angle (randomly selected from 10 to 165 degrees) of the spherical coordinate system. All virtual cameras point toward the centroid of m, and one image is rendered form each camera. The intensities of pixels in the rendered images are determined by interpolating the reflected intensities of the polygon ver-tices. Due to the randomness of the sampled views, some parts of objects would be dark following the traditional settings if only one light source is placed during rendering. To avoid the problem, in our rendering process, two light sources are placed facing each other, while the mesh object is in between. The model shapes are uniformly scaled to fit into the perspective view. Note that V images at different views are rendered for each 3D object, and up to two of the rendered images are used in each input training sample, and v ≤ V images are used in the testing phase.
Point cloud sampling: Following [37], we adopt the Farthest Point Sampling (FPS) algorithm to sample point clouds from each mesh object surface in the mesh datasets. Starting from a randomly chosen point, the next point is sampled in turn according to the average distance to all sampled points, that is, the farthest point. Each mesh object is uniformly sampled 2,048 points to keep the shape information of the object as much as possible. All sampled points are then normalized into a unit sphere.
As illustrated in Figure 2, there are three networks in our framework: a 2DCNN (F img ) to extract 2D features from images cross different views, a graph neural network (F p ) to extract 3D features from unordered point cloud data, and a two-layer fully connected neural network F f to predict the cross-modality correspondence based on the two types of features extracted by F img and F p . The three networks are jointly optimized by cross-modality correspondence, meanwhile, the network F img is optimized by cross-view correspondence (see details in Section 3.3).
The 2D image feature learning network (F img ) employs ResNet18 [14] as the backbone network with four convolution blocks with a number of {64, 128, 256, and 512} 3 × 3 kernels. Each convolution block includes two convolution layers followed by a batch-normalization layer and a ReLU layer, except the first convolution block which consists of one convolution layer, one batch-normalization layer, and one max-pooling layer. A global average pooling layer, after the fourth convolution blocks, is used to obtain the global features for each image. Unless specifically pointed out, a 512-dimensional vector after the global average pooling layer is used for all our experiments.
The 3D point cloud feature learning network (F p ) employs dynamic graph convolutional neural network (DGCNN) [53] as the backbone model due to its capability to model local structures of each point by dynamically constructed graphs and its good performance on classification and segmentation tasks. There are four EdgeConv layers and the number of kernels in each layer is 64, 64, 64, and 128, respectively. Each convolution graph consists of one KNN graph layer which builds the KNN graph for each point and two convolution layers. Each convolution layer is followed by a batch-normalization layer and a leaky ReLU layer. The EdgeConv layers aim to construct graphs over k nearest neighbors calculated by KNN and the features for each point are calculated by an MLP over all the k closest points. After the four EdgeConv blocks, a 512-dimension fully connected layer is used to extract per-point features for each point and then a max-pooling layer is employed to extract global features for each object. The two-layer fully connected neural network F f is employed for cross-modality classification, which consists of a 256-dimensional fully connected layer and a 2-dimensional fully connected layer. Each feature vector feeding into F f is extracted by F img and F p and concatenated together as a 1024-dimension vector. The output of F f is a binary classification value.
In our proposed self-supervised learning schema, two types of constraints are used as supervision signals to optimize the networks: cross-modality correspondence and cross-view correspondence. The cross-modality correspondence requires networks to learn modality-invariant features extracted from two different modalities F img and F p , while the cross-view correspondence requires the subnetwork F img to capture semantic 2D image features to match objects from random views. We formulae the crossmodality task as a classification task and the cross-view task as a metric learning task.
Let D = {sample (1) , ..., sample (N ) } denotes training data of size N . The i-th input sample sample (i) = {p (i) , img
3 }, where p (i) and img
2 represent the point cloud and two different rendered views generated from the same 3D mesh object respectively, and img
3 is an image rendered from a different object. The labels y (i) j ∈ {0, 1} indicates whether the point cloud p (i) and the rendered image img (i) j are from same object where 1 for same object and 0 for different objects. Note that img (i) 1 and img (i) 2 are randomly selected in V rendered views from a 3D mesh object same as the sampled point cloud p (i) , while img
Cross-view correspondence: The objective of the cross-view task is to train the network F img to learn view invariant features from rendered images. When an object observed from different views, the visible parts may look differently, however, the semantic features for images in different views should be similar. Therefore, triplet loss [43] is employed here to train the network to minimize distance of features of positive pairs (i.e. from same object) and maxi-mize distance of features of negative pairs (i.e. from different objects):
where the triple samples img
2 and img
3 correspond to anchor, positive and negative rendered images, α is the margin hyper-parameter to control the differences of intra-and inter-objects.
Cross-modality correspondence: The cross-modality learning is modeled as a binary classification task by employing the cross-entropy loss to optimize all the three networks. After obtaining image features by F img from rendered images and point cloud features by F p from point clouds, the network F f predicts whether the two input data of different modalities are from same object by discovering the high-level modality invariant features. The positive samples are the point cloud and image pairs from same 3D mesh object, while the negative samples are from different objects. The loss function for jointly optimizing networks F img , F p , and F f is:
The input features of F f are extracted by F img and F p , and F f learns the correlation of the features extracted from two different data modalities. When jointly train the three networks, a linear weighted combination of the loss functions L triplet and L cross are employed to optimize the whole framework. The final selflearning loss is combined as:
where β is the weight for the cross-modality loss.
The details of the joint training process are illustrated in Algorithm 1. After the jointly training finished, two networks F img and F are obtained as pre-trained models for two different modalities. The joint training enables the two feature extractors to learn more discriminative and robust features cross different data domains.
Self-supervised learning: The proposed framework is optimized end-to-end using the SGD optimizer with an initial learning rate of 0.001, the moment of 0.9, and weight = Fimg(img
2 ); f i
update networks Fimg, Fp and F f to minimize L end for return pre-trained networks Fimg and Fp decay of 0.0005. The learning rate decreases by 90% every 40, 000 iteration. The networks for self-supervised learning are trained on the ModelNet40 dataset for 120, 000 iterations using a mini-batch size of 32. To learn more robust features, data augmentation is applied to both images and point clouds. The images are randomly cropped and randomly flipped with 50% probability in the horizontal direction, while the point clouds are randomly rotated between [0, 2π] degrees along the up-axis, randomly jittered the position of each point by Gaussian noise with zero mean and 0.02 standard deviation. The rendering views V is 180 for each 3D mesh object in the dataset. During the testing, we randomly select 2D-2D and 2D-3D testing pairs from the test split of ModelNet40 and ModelNet10. The amount of two types of pairs is ten times the test split including half positive pairs and half negative pairs. Evaluation of learned 2D and 3D features: To evaluate the effectiveness and generalization of the learned 2D and 3D features by the proposed self-supervised learning schema, five different tasks are designed as follows. For the multi-view 2D shape recognition and 3D shape recognition tasks, the image and point cloud features are extracted by two pre-trained networks F img and F p , then trained on corresponding SVMs with one class linear kernel, respectively. For the 3D part segmentation task, additional fully connected layers are added on top of the pre-trained F p and then fine-tuned on the ShapeNet [4] dataset. The network is optimized with Adam optimizer [21] using an initial learning rate of 0.003 and decreased by 90% every 20 epochs. For the 2D and 3D shape retrieval tasks, Euclidean distance over the global features of two objects is used as a metric to measure the similarity of two objects. Datasets: All the experiments are conducted on two 3D object benchmarks: ModelNet40 [56] and ShapeNet [4].
The ModelNet40 dataset contains 12, 311 meshed models covering 40 classes, of which 9, 843 are used for training and 2, 468 for testing. The ModelNet40 is used to train our proposed self-supervised learning framework as well as for the evaluation tasks of multi-view 2D shape recognition and 3D shape recognition. The ModelNet10, a subset of ModelNet40, is also used as a testing set, which contains 10 classes. The ShapeNet contains 16 object categories including 12, 137 models for training and 2, 874 for testing and it is employed to evaluate the task of 3D part segmentation. In all experiments, 2, 048 points are sampled for each 3D mesh object as the input point cloud data.
A straightforward evaluation of the effectiveness of our proposed self-supervised learning framework is to recognize the cross-modality and cross-view correspondence with ModelNet40 and ModelNet10 datasets. Table 1 and 2 report the cross-modality recognition accuracy and crossview feature Euclidean distance of testing image pairs. For the cross-modality recognition task in Table 1, our networks accomplish over 90% accuracy which shows that the self-supervised learning successfully learns modality invariant features. For the cross-view correspondence recognition in Table 2, the margins between the mean distance of positive pairs and that of negative pairs are very large which demonstrates that the networks indeed learn the view-invariant features. When the networks trained jointly with cross-modality and cross-view correspondence, although the performance of cross-modality recognition decreases a little bit, the standard deviations for the distances of both positives and negatives are significantly improved (see rows 2 and 4) which validate that the cross-view correspondence enforces the learning of view-invariant features.
One common problem of self-supervised learning is that the network can easily learn trivial features (e.g. corners, edges, or other low-level features) instead of high-level semantic features. To further analyze the features extracted by F img and F p , we use T-distributed Stochastic Neighbor Embedding (TSNE) [30] to visualize the learned 2D and 3D features of the top 10 object categories in ten different colors on ModelNet40 as shown in Fig. 3. Each point indicates one feature that is max-pooled from v extracted features of v views. In the feature space, the features belong to the same class are closer than the features from different object classes, which show that the network indeed can learn high-level semantic features.
Our proposed framework effectively learns both 2D and 3D features and achieves high performance on the pretext task. Here, we further evaluate the learned 2D and 3D features (i.e. F img and F p ) as pre-trained models on other down-stream supervised tasks: 2D and 3D shape recognition on ModelNet40 dataset. Two linear SVM classifiers are trained based on the extracted 2D and 3D features by F img and F p , respectively. Same as in subsection 4.2, each extracted feature for 2D recognition task is max pooled from v extracted features of v random views, except when v = 1.
Table 3. The performance of using the self-supervised learned models as feature extractors on the 2D and 3D shape recognition tasks on the ModelNet40 dataset. Both 2D and 3D shape recognition tasks are benefited from jointly training with cross-view and cross-modality correspondences. When multiple views (#Views = 12, 36, or 80) are available for testing, the performance of 2D shape recognition is significantly improved. As shown in Table 3, both the pre-trained F img and F p can achieve high accuracy on the 2D and 3D shape recognition tasks (89.3% and 89.8%) to recognize 40 object categories on ModelNet40 dataset which show that the two networks learn discriminative semantic features through the
Features from Images of v views Features from Point Cloud self-supervised learning process. When only trained with cross-modality correspondence, the 3D features learned by F p -CM achieve 87.5% accuracy while the performance of 2D features by F img -CM is only 66.1%. The joint training of cross-view and cross-modality correspondence significantly improves the performance of F img on 2D recognition (+6.4%) and F p on 3D recognition (+2.3%). The accuracy of 2D recognition is further boosted by more discriminative image features max pooled from multi-testingview features, achieving 89.3% with 80 views from each data.
To evaluate the generalization ability of the learned features, we further evaluate both 2D and 3D features extracted by F img and F p on shape retrieval tasks on the ModelNet40 dataset and Top-K accuracy are reported in Table 4. Since no other self-supervised learning methods for point cloud or multi-view images have reported performance on this task, we directly compare with ImageNet pre-trained models on the retrieval task. The 3D network F p -CM and F p -CM-CV can accomplish the retrieval task with high accuracy. As for the 2D network F img , the performance is significantly improved when more views are used to represent each object. When only using 1 view for each image, our self-supervised model F img -CM-CV outperforms the ImageNet pre-trained model. When more views (12 or 80) are available, our model achieves comparable performance with the supervised model which is pretrained on the ImageNet dataset.
To further verify the quality of 3D features learning by the pre-trained F p for point cloud data, we conduct the transfer learning on the 3D part segmentation task with the ShapeNet dataset. To adapt F p on the 3D part segmentation task, four fully connected layers are added on the top of F p , and the output from all the four blocks and the global features are used to predict the pixel-wise labels. Three sets of experiments are studied: (1) Only update the four newly added layers with frozen F p , (2) F p and newly added layers are randomly initialized and supervised trained from scratch [37], (3) The learned features by F p are used as pretrained models and all the layers are fine-tuned (unfrozen). The extensive studies of train/fine-tune strategies with different amounts of training data on the ShapeNet dataset for the 3D part segmentation are shown in Table 5.
As shown in Table 5, training with cross-view correspondence can improve the ability of F p to recognize object parts. When 100% of the training data are available, even without updating the parameters of F p on the new task, it still achieves 80.8% instance mIOU which is only 2.2% lower than the supervised model. It validates that F p can learn semantic features from the proposed pretext task and transfer them across datasets and tasks. When the full network is initialized with the pre-training weights and further fine-tuned, the instance mIOU improves by 0.4% and the class mIOU improves by 0.5% showing that the learned weights for F p from self-supervised pretext task can be served as a good starting point for the optimization. When using only 20% data, the parameter-unfrozen setup can significantly (+2.3% on class mIOU, and +1.2% on instance mIOU) boost up the performance than the supervised setup. When using only 2% of the data, the performance of both parameter-frozen setup (+2.2% on class mIOU, and +3.1% on instance mIOU) and parameter-unfrozen setup (+4.5% on class mIOU, and +5.0% on instance mIOU) are better than the supervised setup. Our pre-trained F p performs well when fine-tuned on small-scale 3D shape datasets.
In this section, we further compare our pre-trained F img and F p with the state-of-the-art methods for 3D shape recognition on ModelNet40 dataset including 2D imagebased methods [41,41,46] and 3D methods of both unsupervised learning models [1,5,8,10,13,20,44,54,57,59] and supervised learning models [8,15,23,28,39,45,52,53]. The setups of our models are same as in subsection 4.3. The comparisons are shown in Table 6.
Compared to other unsupervised feature learning methods in Table 6, our approach achieves the state-of-the-art accuracy on the ModelNet40 shape recognition task with pre-trained F p and a linear SVM. The performance of F p trained with both cross-modality and cross-view correspondences is 89.8% which is 0.7% higher than the previous state-of-the-art method. Even trained without using any human-annotated labels, the features learned by our network achieve comparable performance as the supervised methods on the ModelNet40 dataset. Moreover, almost all the existing self-supervised learning methods can learn only 2D image features or only 3D point cloud features, while our method can jointly learn both the discriminative 2D and 3D features that outperform previous state-of-the-art selfsupervised learning methods.
In this paper, we have proposed a self-supervised learning schema that can jointly learn discriminative 2D and 3D features by using the cross-view and cross-modality correspondences on the 3D point cloud datasets. The learned features from both the 2D image-based network and the 3D point cloud-based graph neural network have been extensively tested across different tasks including multiview 2D shape recognition, 3D shape recognition, multiview 2D shape retrieval, 3D shape retrieval, and 3D partsegmentation, showing strong generalization abilities of the learned features. Our results demonstrate a promising direction to learn features by exploiting cross-modality correspondence among different modalities derived from 3D data including mesh, rendered multi-view data, voxel, point cloud, Phong, depth, Silhouette, etc.