In recent years many successful deep learning architectures for 3D geometric deep learning and point cloud anal-ysis in particular have been developed, we refer to [14] for a comprehensive survey. Yet, most methods developed for point cloud analysis are not robust to rotations. Furthermore point cloud dataset like ModelNet [33] or ShapeNet [6] used for training and evaluating these methods often consist of synthetic pre-aligned shapes. In consequence, well established point cloud methods like [25,27,4,31,21] fail to generalise to unseen poses. An important performance gap between the aligned and unaligned or rotation augmented settings has been reported in in multiple publications [11,24,7,37]. Recently numerous works [12,22,24,7,37,36,20,39,28] have addressed the issue of designing rotation invariant deep learning architectures for 3D data analysis with a variety of approaches. A particularly interesting property for 3D deep learning algorithms closely related to rotation invariance is rotation equivariance. At a high level a rotation equivariant neural network produces features that undergo a rotation in feature space given a rotation of the input. The key point is that this transform only depends on the rotation applied to the input. This is a precious property for learning, guaranteeing immediate generalization to unseen poses. Many rotation equivariant designs [30,17,11,32,2] have been proposed, thanks to a well established theory of SO(3) representations. A fundamental challenge in the design of equivariant networks is the design of equivariant non-linearities. To preserve equivariance such non-linearities must commute with the rotations of equivariant features, this constraint limits the possible designs. The aforementioned methods either rely on non-linearities applied to rotation invariant quantities like the norm of equivariant features which cannot capture directional information or, consider polynomial equivariant features which can be expensive to compute and complicates training. In the recent work [9] Hann et al. introduce a non linearity for local intrinsic rotation equivariant features over surfaces by interpreting the features as Fourier coefficients of functions over the circle, composing these functions with non linear activations and getting new equivariant features by computing the Fourier coefficients of the composed functions. In this work we explore a similar ap-1 proach in the context of 3D rotation equivariant features from Tensor Field Networks [30] over point-clouds. We propose a simple yet effective design of rotation equivariant non-linearities for Tensor Field Networks. TFN relies on convolutional filters based on spherical harmonics and the resulting features share the same equivariant properties as the spherical harmonics basis. We interpret such feature vectors as the set of coefficients of a function on the unit sphere. Moving to a sphere function representation by applying inverse spherical harmonics transform we can apply non trivial activation functions or MLP's in a pointwise fashion over the sphere. We then return back to the equivariant feature representation by computing the spherical harmonics transform of the resulting function. This operation has a non trivial effect on both the norm and direction of equivariant features, allowing to capture directional information. We present the general setting of point cloud convolutions and TFN and briefly review the underlying theory in section (2). In section (3) we describe our method and specificities of our design compared to the general approach described in section (2). We present our results in section (4) where we compare our method to state of the art rotation invariant methods for shape classification and segmentation and show improvements over existing equivariant non-linearities. We believe our contibution is of general relevance to other equivaraint networks like [32] sharing similar structures to TFN.
In recent years the field of point cloud processing has developed a lot thanks to many works introducing various notions of graph or point-cloud convolution [27,21,4,31]. Graph convolution aggregates information over the neighbourhood of each vertex. Point cloud and graph convolutions are related as we can define a notion of point cloud convolution by considering the graph of k nearest neighbours of the point cloud. More generally, it is convenient to aggregate information in the neighborhoods of points from a second point cloud, typically a sub-sampled or an upsampled point cloud, similarly to the idea of strided convolutions. The general formulation for graph convolutions is as follows: Let V be the set of vertices and X , Y ⊂ V non empty subsets of V. Given signals x : X → R k , y : Y → R k and f : X → R and a relation kernel function κ : X × Y → R, the convolution of f by κ is the signal g : Y → R defined by:
where is a permutation invariant operator like or max, j ranges across the neighbours in X of vertex i in Y and c i is a normalization factor. We follow an approach similar to [4,30,24] for point cloud convolution where, x : X → R 3 , y : Y → R 3 are the points euclidean coordinates of point clouds X and Y and, κ : R 3 → R is a kernel function:
where j ranges across the k-nearest neighbours of i in Y or points in Y within a ball of a given radius centred at i. A typical choice for c i is to divide by the number of neighbours of i, c i = 1 |N (i)| but in practice we will choose a different normalization as we shall see later. In deep learning applications a collection of kernel functions κ m is chosen and linear combinations of the f * x,y κ m are learned. Feature propagation We also borrow the feature propagation layers from [27] for segmentation tasks. Feature propagation allows to transfer signals between two point clouds. Given two point clouds x and y and a signal f over x, the feature propagation of f to y is defined as follows:
where j ranges across the 3 nearest neighbors of x i in y. Feature propagation is used in the segmentation architecture of [27] which is a variant of U-Net architecture. The network computes series of conv like layers down sampling the signal, then feature propagation layers up sample the signal to concatenate it with the output of the conv like layer of corresponding resolution and, applies a fully connected layer to the channels axis followed by a batch normalization layer and a ReLu activation. An MLP is applied to the last layer for final point classification.
In this subsection we describe the construction of 3D rotation equivariant CNNs and we refer to [18,8,16] for a more extensive view of the underlying theory. We especially consider Tensor Field Networks [30] which operate on point clouds but, our description also partly covers other works like [32,17,2] which also rely on equivariant convolution based on steerable kernels. 3D Rotation equivariant CNN's are built around the notion of 3D steerable kernels. A steerable kernel basis is a basis of the form:
where
is the vector of degree ℓ ∈ N spherical harmonics and (ϕ r : R → R) r are the radial component of the kernels. A key property of Y ℓ is that it undergoes a rotation in R 2ℓ+1 given a rotation of its input. More precisely for any rotation R ∈ SO(3), there exist a rotation matrix D ℓ (R) ∈ SO(2ℓ + 1) associated to R called the Wigner matrix (of type ℓ) [8,16,18] such that, Y ℓ (Rx) = D ℓ (R)Y ℓ (x). By construction the kernel basis from Eq. ( 4) satisfies the same equivariance property as spherical harmonics:
Many choices are possible for the radial component, we detail our exact kernel design in section (3). Given a point cloud x ∈ R n×3 and a signal f ∈ R n×k over x, we say that v(x, f ) ∈ R 2ℓ+1 is a type ℓ equivariant feature if:
Type ℓ vector features can be global or point-wise. Type 0 features are rotation invariant scalars. Any rotation invariant descriptor or pose independent signal can be thought of as a type 0 feature. Type 1 features undergo rotations in R 3 (up to permutation of axes), the yzx coordinates of the points are type 1 features. The convolution of a type 0 features with a type ℓ kernel produces type ℓ features:
Stacking convolutions with equivariant kernels involves computing convolutions between equivariant kernels and equivariant features of various types. The resulting features are linear combinations of tensor products of equivariant features. The Peter-Weyl theorem [16] implies that such tensor products can be decomposed into orthogonal sums of irreducible features of different types. More specifically, for any k, ℓ and any J ∈ m -ℓ, m + ℓ there exists an orthogonal projection matrix Q (k,l),J such that Q (k,l),J u ⊗ v is a type J feature vector. The convolution of a signal f ℓ of type ℓ vectors over x by a type k equivariant kernel κ rk is the collection of tensors:
) For J ∈ |l -k|, l + k , the coefficients of the matrices Q (k,ℓ),J are given by the so called Clebsch-Gordan coefficients [8]. TFN consist of a multi-head architecture where each head is a tensor stacking pointwise equivariant features of a given type. An equivariant convolution layer takes a collection (f ℓ ) ℓ of such tensors as input, computes their convolution with kernels of different types, decomposes the results into irreducible features and, groups them by type and learns linear combinations between the resulting features:
In practice we only take decomposed features up to type 3. A bias term b can be added for type 0 outputs since the bias is pose independent it is only compatible with type 0 (invariant) features. Notice that only linear combinations between vector features are taken. The reason is that the weights matrices must commute with the Wigner matrix. By a corollary of Schur's lemma [16], a linear map commutes with the Wigner matrix iff it is an homothety. Thus to preserve equivariance we can only take linear combinations of equivariant features of the same type and not between their coordinates. This is a limitation compared to traditional non-equivariant filters where we can consider any linear combinations of the basis filters.
Equivariant Activations A major limitation in the design of rotation equivariant neural networks is the choice of activation functions. To preserve equivariance, the activation function must commute with the Wigner matrix. Equivariant non-linearities follow two trends. Works like [30,32] multiply equivariant features by non-linear functions of invariant features. The resulting features still commute with the Wigner matrix as only their norm is affected and thus they are equivariant. However such non-linearities don't have effect on the direction of features which limits their expressivity. An other direction explored by [17] and [2] is to build equivariant features as polynomials of other equivariant features. While it has been proven in [10] that equivariant polynomial features can approximate equivariant functions, in practice they can be computationally heavy and are difficult to train as reported in section (8.1) of [2]. We now briefly review non-linearities from [30,32,17] and [2]. TFN [30] introduce the notion of norm non-linearities, for a multi head equivariant signal f it is defined by:
Where the ξ is the ReLu function and the b ℓ 's are learnable bias terms. 3D steerable cnns [32] introduce a similar but slightly more elaborate approach called gated nonlinearities. Instead of only producing a multi head equivariant signal f like in TFN, a layer also produces a rotation invariant signal γ ℓ for each equivariant head f ℓ with ℓ > 0 (one can simply split the invariant head of the signal in two parts) the gated non linearity is defined by:
where σ is the sigmoid function and the b ℓ are learnable bias vectors. On the other hand [17] and [2] consider tensor product non-linearities. The idea is to take tensor products of equivariant features and decompose them using the Clebsch Gordan Decomposition producing quadratic equivariant features:
where the ⊕ sign indicates concatenation on the last axis.
The supplementary material of [32] also provides a description of the aforementioned non-linearities.
In this section we present our method and the specific changes we introduce to the pipeline presented in section (2). Our main contribution is to interpret rotation equivariant features as coefficients of functions in the spherical harmonics basis with two applications: Equivariant nonlinearities and equivariant spatial max pooling. Before detailing these points we specify our kernel design and point cloud sampling strategy. Our kernels and sampling strategy As explained in equation (2) of section (2) point cloud convolution requires two point clouds a first point cloud caries the signal and the second one (typically a sub-sampling or up-sampling of the first one) defines the locations at which to apply the convolution kernels. We adopt the same sampling strategy as [24], we consider point clouds whose cardinal is a power of two and index them by a balanced kd-tree, we can then sub-sample the point cloud by applying 1D average pooling with a pool size which also a power of two. This computes the means of sub-trees at a given depth. For convolutions we use a similar kernel to [30,32,24] but our normalization differs slightly as we empirically observed a slight improvement with the L2 normalization (14). Like [32,24] we chose Gaussian shell functions for the radial component functions ϕ of equation ( 4) but, we divide by the sum of the radial functions so that each point in the kernel support is assigned a mass of 1 and, we restrict the kernel support to a ball of radius r. Denoting r j := jr d we define the radial component of the steerable kernel of Eq. ( 4) by:
) We normalize the kernels at each point by their average L2 norm, leading to the following formula for the normalization factor c irℓ in formula (4):
Equivariant features as functions The convolution layers presented in equations (4, 8, 9) of section (2) produce equivariant features tensors (f ℓ ) ℓ where the type ℓ ranges between 0 and ℓ max 0, we set the same number of channels for each head. We observe that type ℓ rotation equivariant features as defined in equation ( 6) of section (2) and spherical harmonics of degree ℓ share the same equivariance properties, that is they are multiplied by the Wigner matrix D ℓ (R) when their input is rotated by R ∈ SO(3). This motivates us to interpret them as coefficients of functions in the spherical harmonics basis. For each point i and channel index c we can compute the inverse Spherical Harmonics Transform (SHT) resulting in a function on the unit sphere:
) Equivariant non-linearities The inverse SHT functional representation allows us to apply non-linearities in a pointwise fashion over the sphere by composing F + (f ) with any activation function ξ and, compute the coefficients of the composition in the spherical harmonics basis using forward SHT: We refer to the supplementary material for a proof. Unfortunately there is no closed form for the integral of equation ( 16) so, in practice we have to rely on a discrete approximation of the SHT and its inverse. For that we consider a discrete sampling of the unit sphere given by a finite set of points p 1 , . . . , p N ∈ S 2 , we define the discretized inverse SHT by:
We then apply the non linearity ξ at all points (p j ) j and compute the coefficients of this discrete approximation of F + (f ) ic into the spherical harmonic basis using the discretized SHT (18). Given a discrete signal {F ijc } over the points P = (p j ) j for all point i of the input point cloud and all channel index c we define our SHT layer by:
We use the the intermediate representation F + (f ) to apply pointwise operations at each sample p j like a dense layer over the channel axis and non-linearities or more generaly MLP's while maintaining equivariance up to our discretization. We apply our activation function ξ to f by:
We consider different samplings of the sphere, a first choice motivated by theorem (3.2) is to use the vertices of polyhedrons with non trivial symmetry groups. Specifically we chose the Regular dodecahedron with its 20 vertices and the Pentakis decahedron with its 32 vertices.
We refer to the supplementary material for a proof. We also consider the Fibonaci Sampling from [23] providing a more systematic and flexible sampling allowing to choose the number of points. Equivariant pooling An other benefit of our approach is that we can use our functional representation to perform max pooling. Given a sub-sampling y of the point cloud x we define local max pooling by:
where j ranges across the k-nearest neighbours of y i in x.
We also define global max-pooling by:
In our experiments we consider a classification and a segmentation architecture. The basic brick of our architectures is our equivariant point cloud convolution layer described in figure (1). It has been shown in [10] that simply concatenate each conv layer of TFN with the yzx coordinates of the underlying point clouds without applying non-linearities is enough to ensure that the resulting class of networks is a universal approximator of equivariant features (up to certain choices of radial functions in the definition of the kernels). Here we adopted this construction to compensate for the loss of information induced by the truncation of the features type. We adopt following notations to describe our layers and network architecture: We denote our equivariant convolution layer by EC(K, r, d, ℓ max , [u 0 , . . . , u k ]) where:
• K is the number of centroid points at which to apply equivariant kernels.
• r is the radius of the K convolution windows.
• d is the number of shells of the kernels.
• ℓ max is the highest equivariant output type of the layer.
• u 0 is the number of units of the dense layers applied to the output of the equivariant convolution.
• [u 1 , . . . , u k ] are the number of units of the MLP layers.
Figure 1. Our convolution layer takes a dictionary of tensors of equivariant features (f ℓ,k ) ℓ,k for ℓ ∈ 0, ℓ k max as input. First we concatenate type 1 features with the yzx coordinates of the underlying point cloud, then we perform equivariant convolution ( 9) and again concatenate type 1 output features with the yzx coordinates of the next point cloud. We apply inverse SHT from Eq. ( 17) followed by a batch norm layer and ReLu activation optionally followed by an MLP. Finally we obtain a new set of equvariant features (f ℓ,k+1 ) ℓ,k for ℓ ∈ 0, ℓ k+1 max by computing SHT from Eq. ( 18).
We denote by F C(k, d) a fully connected layer with k units followed by a batch normalization and ReLu layer and a dropout layer with dropout rate d. Our classification architecture is as follows: EC(256, 0.2, 3, 3, [32,32,32]) → EC(64, 0.4, 3, 3, [64, 64, 64]) → EC (16, 0.8, 3, 3, [128, 128, 256])
It takes a point cloud with 1024 points index it by a kdtree and computes coarser point clouds of size 256, 64 and 16 using average kd-tree pooling. Then it computes three successive equivariant convolution layers as in figure (1), taking the the constant signal equal to 1 as a type 0 feature input to the first layer. After the last convolution we have equivariant tensors of types 0 to 3 representing a signal over the 16 points of the last point cloud, we apply a global max pooling layer (21), compute the norms of the resulting global features and pass them through a final two layers MLP for the class prediction. For segmentation we adopt a U-Net like architecture inspired by the segmentation architecture of [27]. Like our classification architecture we have three equivariant convolution layers going from a point cloud with 2048 points to a coarse point cloud of 32 points. We then up sample the signal back to 2048 points using feature the propagation layers from [27] described in equation ( 3). The output of each feature propagation layer is concatenated with the output of the convolutional layer on the corresponding point cloud if available or the yzx coordinates of the input point cloud for the last feature propagation layer. Then we apply fully connected layers over the channel axis and equivariant activation as described in figure (1) (with no MLP to reduce computational cost). We denote our feature propagation layer by F P (d) where d is the number of units of the fully connected layers. Our segmentation architecture is as follows:
We evaluate our method in point cloud classification and segmentation tasks. We consider three different experimental train / test settings: z/z, z/SO(3) and SO(3)/SO (3) where z stands for aligned data augmented by random rotations around the vertical axis and SO(3) indicates data augmented by random SO(3) rotations. The data is randomly rotated at each training iteration. We consider different non-linearities for TFN: 'ReLu' where we simply use a ReLu activation in our equivariant convolution layer (EC) described in Figure (1), 'mlp' where we stack a two layer mlp on top of the ReLu activation, 'norm' which is the original TFN non linearity and 'gated' wich is the gated non linearity from [32]. For each non linearity x we denote TFN[x] the TFN version based on x. Thus TFN[norm] is similar to the original TFN. We use the following abbreviations to refer to the sampling pattern: R for the regular dodecahedron, P for the Pentakis dodecahedron and F for Fibonacci sampling. We compare to two categories of methods: rotation sensitive and, rotation robust methods which are specifically designed for handling non aligned data. The z/z case offers a baseline for rotation sensitive methods. The z/SO(3) setting allows to measure the robustness of a method to the observation of new poses unseen during training while, the SO(3)/SO(3) is the most practical setting as in practice it is more representative of real world where the pose of objects is not known in advance nor consistent and, it is always possible to apply random rotation augmentation. For classification we consider two datasets. The Mod-elNet40 dataset [33] consists of 12,311 synthetic shapes in canonical pose divided into 40 categories with a 9,843/2,468 train / test split and the ScanObjectNN dataset [3] which consists of 2902 point clouds sampled from realworld objects across 15 different categories with a 2309/581 train test split. For segmentation we consider the ShapeNet part dataset [6] which contains 16,881 shapes from 16 classes, annotated with 50 parts labels in total. We trained all our models with a TensorFlow [1] implementation using the Adam optimizer [15] with learning rate 0.001. We used a batch size of 32 for classification reported in section (4.1) and 24 for segmentation reported in section (4.2).
Table (1) shows our results for classification on Mod-elNet40 and compares it to state of the art methods. We considered an alternative variant of our method using prealignment with PCA refereed to by the indication (PCA). We trained our models for 200 epochs and observed convergence in around 150 epochs. We report two different metrics, since the ModelNet40 dataset does not have an official validation set we used the test set as validation set and select the best model over training based on validation score. We average the test score over 40 randomly rotated copies of the test set to smooth the variability introduced by random rotations augmentation. We use the same best z/z model for both the z/z and z/SO(3) scores. We refer to these results with the indication (best). We observed oscillations of validation accuracy even after convergence we included these results to indicate an upper bound of the performance of our models. For fair comparison we also report the average validation accuracy over the last 10 epochs which is more representative of actual performance. While our method is mostly robust to rotations as suggested by the z/SO(3) setting, we still observe a performance drop but this is due to our pooling and activation being only approximately equivariant in practice. We notice that our PCA pre-aligned version produced more consistent results across the 3 different settings z/z, z/SO(3) and SO(3)/SO(3). Our method does not quite match the best state of the art methods yet, these methods either rely on richer input data or slightly different representation. EMVnet [12] consider a multi-view image based representation of the shapes based on renderings of meshes. LGR-Net [39] relies on surface normals, SFCNN [28] propose an original approach similar to multi-view by mapping input point-clouds to a sphere. Our results are close to GC-Conv [36] and RI-Framework [20] which both rely only on the point coordinates. A fundamental question is whether it is enough to stack purely invariant layers for rotation invariant tasks, it has been stated in in GC-Conv [36] that rotation invariant features are usually not as distinctive as non invariant ones. In this work we rely on equivariant layers which unlike invariant layers can transmit directional information to the next layers. Our baseline for rotation invariant layers is SPHNet [24] as we use very similar spherical harmonics based kernels in a 3 layered conv architecture, the difference is that SPHNet takes the norm of each equivariant features right after convolution instead of computing equivariant convolution. These observation together with our results showing a noticeable improvement over SPHNet suggest that simply stacking invariant layers is not optimal. We propose an ablation study of our ModelNet40 experiment in table (2) to compare our equivariant non linearity (19) to the norm non linearity (11) introduced in [30] and gated non linearity (12) introduced in [32]. We also study the impact of the number of samples and the sampling pattern on the performance of our non linearity. For better comparison with other non-linearities we consider TFN[ReLu] so that we only measure the effect of our new non linearity. We apply the norm and gated non linearites on top of the fully connected layers following the equivariant convolution step. To produce a global descriptor layers in our classification architecture we apply norm in a pointwise fashion, reducing equivariant features to invariant scalars, allowing to apply rotation invariant global max pooling. Results are shown in table 2, we observe a noticeable improvement over the norm and gated non-linearities baselines and interestingly the results of TFN[norm] are very close to the results of SPHNet [24] shown in table (1). Although the later uses purely invariant layers both rely on non similar linearities applied only to the norm of equivariant features. As for the effect of the number of samples we observe as expected that more samples improves robustness to rotations as shown by the z/SO(3) setting. We add a qualitative study of the equivariance error w.r.t. the number of samples in the supplementary material. We report results on ScanObjectNN [3] in table (4). We compare our method to several state of the art methods, in particular the recent SE(3)-Transformers framework [13] introducing a transformer architecture built on top of TFN. We follow the same experimental setting as [13], training our network until convergence (100 epochs) and average over 5 runs on the basic OBJ ONLY version of the dataset with no rotation augmentation and using the z coordinate as a 0 type input (indicated by +z). We achieve comparable results to SE(3)-Transf.+z [13] and noticeably better results than TFN+z. We note that SE(3)-Transf uses less input points but in (Appendix D.1.2) of [13] the authors report that increasing the number of point did not improve the performance. We also adapted the radius of our kernels(0.14, 0.28, 0.56 instead of 0.2, 0.4, 0.8 for Modelnet40.
For our segmentation experiment on ShapeNet [6] we used the official train/val/test split we select the best model according to the the validation accuracy over a 150 epochs training which is a fair metric since validation and test sets are distinct. Table (3) compares our per class and average class intersection over union scores (IoU) to state of the art methods. Our method falls in between GC-Conv [36] and RI-Framework [20] which both use only xyz coordinates of the point clouds as we do unlike the best performing method LGR-Net [39] which also uses normals. We again observe a noticeable improvement over other TFN non-linearities.
Rot-sensitive Methods z/z z/SO(3) SO(3)/SO(3)
SubVolSup MO [26] 89.5 45.5 85.0 PointNet [25] 87.0 21.6 80.3 PointNet++ [27] 91.8 28.4 85 PointCNN [21] 92.5 41.2 84.5 DGCNN [31] 92.9 20.0 81.1 ShellNet [38] 93.1 19.9 87.8 PCNN [4] 92.3 11.9 85.1 Rot-robust Methods z/z z/SO(3) SO(3)/SO(3)
EMVnet [12] 94.4 -91.1 SPHNet [24] 87.7 86.6 87.6 Spherical-CNN [11] 88.9 76.7 86.9 a 3 S-CNN [22] 89.6 87.9 88.7 ClusterNet [7] 87.1 87.1 87.1 RI-Conv [37] 86.5 86.4 86.4 GC-Conv [36] 89.0 89.1 89.2 RI-Framework [20] 89.4 89.4 89.3 LGR-Net [39] 90.9 90.9 91.1 SFCNN(xyz) [28] 91.4 84.8 90.1 SFCNN(xyz+n) [28] 92. Methods # points test acc DeepSet [35] 1024 71.4 3DmFV [5] 1024 73.8 Set Transformer [19] 1024 74.1 PointNet [25] 1024 79.2 SpiderCNN [34] 1024 79.5 TFN+z [30] 128 81.0 PointNet++ [27] 1024 84.3 SE(3)-Transf.+z [13] 128 85.0 PointCNN [21] 1024 85.5 DGCNN [31] 1024 86.
In this work we highlight the importance of the design of equivariant non-linearities. We introduced a novel rotation equivariant non linearity for Tensor Field Networks [30] based on a functional interpretation of equivariant features. In future work we would like to explore this idea further. An interesting direction would be to use equivariant features to produce rotation equivariant functions over the 3D space with potential applications such as rotation equivariant auto encoder based on implicit surface representation. In our experiments we observed that better performing methods such as [39,28] often use point cloud normals. Tensor Field Networks are direction agnostic as they are SO(3) equivariant. Adding local direction information like normals allows to reduce the dimension of the rotation group by restricting SO(3) tor rotations around the normals which might help designing more informative and cost effective equivariant non-linearities.