Determining the pose of an object from an image is a challenging problem with important applications in artificial reality, robotics, and autonomous vehicles. Traditionally, pose estimation has been approached as a point regression problem, minimizing the error to a single ground truth 3D rotation. In this way, object symmetries are manually disambiguated using domain knowledge (Xiang et al., 2018) and uncertainty is not accounted for. This approach to pose estimation cannot scale to the open-world setting where we wish to reason about uncertainty from sensor noise or occlusions and model novel objects with unknown symmetries.
Recent work has instead attempted to learn a distribution over poses. Single rotation labels can be modeled as random samples from the distribution over object symmetries, which removes the need for injecting domain knowledge. For instance, a table with front-back symmetry presents a challenge for single pose regression methods, but can be effectively modeled with a bimodal distribution. The drawback to learning distributions over the large space of 3D rotations is that it requires lots of data, especially when modeling hundreds of instances across multiple object categories. This poor data efficiency can be improved by constraining the weights to encode symmetries present in the problem (Cohen & Welling, 2016). The pose prediction problem exhibits 3D rotational symmetry, e.g. the SO(3) abstract group. That is, if we change the canonical reference frame of an object, the predictions of our model should transform correspondingly. For certain input modalities, such as point clouds or 3D camera images, the symmetry group acts directly on the input data via 3D rotation matrices. Thus, many networks exploit the symmetry with end-to-end SO(3) equivariance to achieve sample efficient pose estimation. However, achieving 3D rotation equivariance in a network trained on 2D images is less explored. Thus, we present Image2Sphere, I2S, a novel method that learns SO(3)-equivariant features to represent distributions over 3D rotations. Features extracted by a convolutional network are projected from image space onto the half 2-sphere. Then, spherical convolution is performed on the features with a learned filter over the entire 2-sphere, resulting in a signal that is equivariant to 3D rotations. A final SO(3) group convolution operation produces a probability distribution over SO(3) parameterized in the Fourier domain. Our method can be trained to accurately predict object orientation and correctly express ambiguous orientations for objects with symmetries not specified at training time.
I2S achieves state-of-the-art performance on the PASCAL3D+ pose estimation dataset, and outperforms all baselines on the ModelNet10-SO(3) dataset. We demonstrate that our proposed architecture for learning SO(3) equivariant features from images empirically outperforms a variety of sensible, alternative approaches. In addition, we use the diagnostic SYMSOL datasets to show that our approach is more expressive than methods using parametric families of multi-modal distributions at representing complex object symmetries.
• We propose a novel hybrid architecture that uses non-equivariant layers to learn SO(3)equivariant features which are further processed by equivariant layers. • Our method uses the Fourier basis of SO(3) to more efficiently represent detailed distributions over pose than other methods. • We empirically demonstrate our method is able to describe ambiguities in pose due to partial observability or object symmmetry unlike point estimate methods. • I2S achieves SOTA performance on PASCAL3D+, a challenging pose estimation benchmark using real-world images.
6D Pose Estimation Predicting the 6D pose (e.g. 3D position and orientation) of objects in an image has important applications in fields like robotics (Tremblay et al., 2018), autonomous vehicles (Geiger et al., 2012), and microscopy (Levy et al., 2022). Most popular methods use deep convolutional networks, which are robust to occlusions and can handle multi-object scenes with a segmentation module (He et al., 2017). Convolutional networks have been trained using a variety of output formulations. Xiang et al. (2018) regresses the 3D bounding box of the object in pixel space, while He et al. (2020) predicts 3D keypoints on the object with which the pose can be extracted. Another line of work (Wang et al., 2019;Li et al., 2019;Zakharov et al., 2019) outputs a dense representation of the object's coordinate space. Most of these methods are benchmarked on datasets with limited number of object instances (Hinterstoisser et al., 2011;Xiang et al., 2018). In contrast, our method is evaluated on datasets that have hundreds of object instances or novel instances in the test set. Moreover, our method makes minimal assumptions about the labels, requiring only a 3D rotation matrix per image regardless of underlying object symmetry.
Rotation Equivariance Symmetries present in data can be preserved using equivariant neural networks to improve performance and sample efficiency. For the symmetry group of 3D rotations, SO(3), a number of equivariant models have been proposed. equivariant embeddings from image input for object pose prediction; however, they use a supervised loss to replicate the embeddings of a spherical convolutional network pretrained on 3D images. In contrast, our method incorporates a novel architecture for achieving SO(3) equivariance from image inputs that can be trained end-to-end on the challenging pose prediction tasks.
Uncertainty over SO(3) Due to object symmetry or occlusion, there may be a set of equivalent rotations that result in the same object appearance, which makes pose prediction challenging. Most early works into object pose prediction have avoided this issue by either breaking the symmetry when labelling the data (Xiang et al., 2014) or applying loss functions to handle to known symmetries (Xiang et al., 2018;Wang et al., 2019). However, this approach requires knowing what symmetries are present in the data, and does not work for objects that have ambiguous orientations due to occlusion (e.g. coffee mug when the handle is not visible). Several works have proposed models to reason about orientation uncertainty by predicting the parameters of von Mises (Prokudin et al., in terms of the spherical harmonics Y l k and for signals over SO(3) in terms of Wigner D-matrix coefficients D l mn . Writing f : SO(3) → R in terms of the D l mn and then truncating to a certain frequency l ≤ L we obtain an approximate representation f (g) ≈ L l=0 2l+1 m=0 2l+1 n=0 c l mn D l mn (g). SO(3) group convolution (2) can be efficiently computed in the Fourier domain using the convolution theoremfoot_0 . Namely, the convolution of two functions is calculated as the element-wise product of the functions in the Fourier domain. For functions over SO(3), the Fourier domain is described by the Wigner D-matrix coefficients, corresponding to a block diagonal matrix with a (2l + 1) × (2l + 1) block for each l (Knapp, 1996). A functions over S 2 in the Fourier domain is described by a vector of coefficients of spherical harmonics and convolution is performed using an outer product (Cohen et al., 2018a). This generates the same block diagonal matrix output as SO(3) convolution (e.g. both S 2 and SO(3) convolution generate signals that live on SO(3)). See Geiger et al. (2022) for efficient implementation of Fourier and inverse Fourier transforms of S 2 and SO(3) signals.
The output of the above group convolution is a signal f : SO(3) → R defined as a linear combination of Wigner D-matrices. While this signal is useful for efficient group convolution, it cannot be directly normalized to produce a probability distribution. Instead, the signal can be queried at discrete points in SO(3) and then normalized using a softmax. To reduce errors introduced by the discretization, points should be taken from an equivolumetric grid. Equivolumetric grids over SO(3) can be generated using an extension of the HEALPix method developed by Yershova et al. (2010). The HEALPix method (Gorski et al., 2005) starts with 12 'pixels' that cover the sphere, and recursively divides each pixel into four sections. The latitude and longitude of each pixel describe two Euler angles of a rotation. The extension to SO(3) specifies the final Euler angle in a way that the grid is equally-spaced in SO(3). The resolution of the final grid can be controlled by the number of recursions.
Our method, Image2Sphere (I2S), is designed to predict object pose from images under minimal assumptions. Much previous work assumes objects with either no symmetry or known symmetry, and thus it is sufficient to output a single point estimate of the pose. Our method, in contrast, outputs a distribution over poses, which allows us to represent uncertainty due to partial observability and the inherent ambiguity in pose resulting from object symmetries. Recent work using distribution learning has suffered from difficulty in training multi-modal distributions and high data requirements. I2S circumvents these challenges by reasoning about uncertainty in the Fourier basis of SO(3), which is simple to train over and allows us to leverage equivariant layers for better data efficiency.
The I2S network consists of an encoder, a projection step, and a 2-layer spherical pose predictor (see Figure 1). I2S generates SO(3) equivariant features from traditional convolutional network encoders. Our method then maps features from image space to the 2-sphere using an orthographic projection operation. Then, we perform two spherical convolution operations. Importantly, the output of the spherical convolution is a signal over the Fourier basis of SO(3), which can be queried in the spatial domain to create highly expressive distributions over the space of 3D rotations.
We use orthographic projection to map the output image feature map f : R 2 → R h from the ResNet to a spherical signal Ψ : S 2 → R h . Orthographic projection P : S 2 → R 2 defined P (x, y, z) = (x, y) maps a hemisphere onto the unit disk. By positioning the feature map f around the unit disk before projecting, we get a localized signal Ψ(x) = f (P (x)) supported over one hemisphere. The advantage of this method is that it preserves the spatial information of features in the original image. Practically, to compute Ψ, a HEALPix grid is generated over a hemisphere {x i } ⊂ S 2 and mapped to positions {P (x i )} in the image feature map. The value of Ψ(x i ) is interpolated from the value of f at pixels near position P (x i ). This process is illustrated in Figure 2.
To demonstrate the strengths of our method, we evaluate it on several challenging object orientation estimation datasets. Additional details can be found in Appendix B.2.
The first dataset, ModelNet10-SO(3) (Liao et al., 2019), is composed of rendered images of synthetic, untextured objects from ModelNet10 (Wu et al., 2015). The dataset includes 4,899 object instances over 10 categories, with novel camera viewpoints in the test set. Each image is labelled with a single 3D rotation matrix, even though some categories, such as desks and bathtubs, can have an ambiguous pose due to symmetry. For this reason, the dataset presents a challenge to methods that cannot reason about uncertainty over orientation.
Next, PASCAL3D+, (Xiang et al., 2014), is a popular benchmark for pose estimation that includes real images of objects from 12 categories. The dataset labels symmetric object categories in a consistent manner (e.g. bottles are symmetric about the z-axis, so label zero rotation about the z-axis), which simplifies the task. To improve accuracy, we follow the common practice of augmenting the training data with synthetic renderings from Su et al. (2015). Nevertheless, PASCAL3D+ still serves as a challenging benchmark due to the high variability of natural textures and presence of novel instances in the test set.
Lastly, the SYMSOL dataset was recently introduced by Murphy et al. ( 2021) to evaluate the expressivity of methods that model distributions over 3D rotations. It includes synthetic renderings of objects split into two groups: geometric objects like the tetrahedron or cylinder with complex symmetries (SYMSOL I), and simple objects with single identifying feature such that the pose is ambiguous when the feature is occluded (SYMSOL II). The dataset provides the full set of equivalent rotation labels for each image so methods can be evaluated on how well they capture the distribution. Note that Murphy et al. ( 2021) generates results using 100k renderings per shape, which we report as a baseline, but the publicly released dataset that we train on only includes 50k renderings per shape.
The goal of pose prediction is to minimize the angular error between the predicted 3D rotation and the ground truth 3D rotation. Two commonly used metrics are the median rotation error (MedErr) and the accuracy within a rotation error threshold (e.g. Acc@15 is the fraction of predictions with 15 degrees or less rotation error). However, these metrics assume that there exists a single, ground truth 3D rotation, which is not valid for symmetric objects or images with pose ambiguity. For ModelNet10-SO(3) and PASCAL3D+, only a single rotation is provided so these metrics must be used. However, when the full set of equivalent rotation labels are provided, like with SYMSOL, a more informative measure is the average log likelihood, computed as the expected log likelihood that the model, p θ , assigns to rotations sampled from the distribution of equivalent rotations p GT : E R∼p GT [log p θ (R|x)]. Achieving high log likelihood requires modelling all symmetries of an object.
I2S uses a residual network (He et al., 2016) with weights pretrained on ImageNet (Deng et al., 2009) to extract dense feature maps from 2D images. We use a ResNet50 backbone for ModelNet10-SO(3) and SYMSOL, and ResNet101 for PASCAL3D+. The orthographic projection uses a HEALPix grid with recursion level of 2, out of which 20 points are randomly selected during each forward pass. We parameterize the learned S 2 filter in the Fourier domain, e.g. learn weights for each spherical harmonic. The filter in the SO(3) convolutional layer is locally supported over rotations up to 22.5 degrees in magnitude. We query the signal in the spatial domain using SO(3) HEALPix grids with 36k points (7.5 degree spacing) during training and 2.4M (1.875 degree spacing) during evaluation. We use the same maximum frequency (L = 6) on all datasets. Additional details on the architecture can be found in Appendix B.1.
I2S is instantiated with PyTorch and the e3nn library (Geiger et al., 2022). It is trained using SGD with Nesterov momentum of 0.9 for 40 epochs using a batch size of 64. The learning rate starts at 0.001 and decays by factor of 0.1 every 15 epochs.
One of the strengths of our method is the ability to represent distributions over 3D rotations. As shown in the previous section, this formulation is beneficial when training on symmetric objects. In this section, we quantitatively evaluate the ability to model uncertainty in two settings: SYMSOL I which includes simple geometric objects with complex symmetries and SYMSOL II which has objects marked with a single identifier such that self-occlusion creates pose ambiguity. Because most images correspond to a set of equivalent rotation labels, we measure performance using average log likelihood, reported in Table 3.
The results show that our method effectively represents complex distributions over SO(3). We achieve higher log likelihood on all SYMSOL shapes than methods that map to a specific small family of distributions such as von Mises or Bingham. This highlights the advantage of parametrizing uncertainty in the Fourier basis of SO(3), which is a simpler approach that avoids training multimodal distributions. We note that Murphy et al. ( 2021) outperforms our method when trained on 100k images per object; however, our method is better when trained on only 10k images per object. This demonstrates an important distinction between the two approaches: our method explicitly encodes the 3D rotation symmetry of the problem in the spherical convolutions, whereas Murphy et al. ( 2021) must learn the symmetry from data. We argue that sample efficiency is important for real world applications since it is not practical to collect 100k images of an single object to model its symmetry.
We want to highlight that we use the same maximum frequency L for all experiments in this work which shows that I2S can be deployed without knowing if object symmetries are present in the task.
We argue that a main driver of our method's pose accuracy is SO(3)-equivariant processing. While many existing methods for end-to-end SO(3)-equivariance have been explored in the literature (Fuchs et al., 2020;Deng et al., 2021;Cohen et al., 2018a), it is not well-understood how to combine non-equivariant and equivariant layers in a network. In this section, we consider other sensible approaches to map from features in the image plane to features that live on SO(3) or a discrete subgroup of it. The approach taken by I2S is to perform orthographic projection to link features on the sphere to the image plane (spatial projection), then convolve it with a filter that is parametrized A DETAILED RESULTS
We compare the sample efficiency of pose prediction methods by training on ModelNet10-SO(3) with limited training views in Table 5. Specifically, we reduce the number of training views from 100 per instance to 20 per instance. Our method outperforms the baselines by a larger margin in the low-data setting. This results highlights an important distinction between our and the baselines: our method explicitly encodes the 3D rotation symmetry present in the pose prediction problem, whereas other methods must learn the symmetry from data.
Here, we include details for baselines that we implemented ourselves. For information on other baselines, refer directly to their work. Zhou et al. (2019) proposes using the Gram-Schmidt process to convert a 6D vector into a valid, 3x3 rotation matrix. To implement this network, we used a ResNet architecture with spatial pooling on the final feature map. The resulting vector was processed with two linear layers to generate the 6D vector. The model is trained with an L2 loss function using a ground truth rotation matrix. The method is trained on all classes at once, but uses a separate linear layer to predict the rotation of each class.
Brégier (2021) proposes using the Procrustes method to convert a 9D vector into a valid, 3x3 rotation matrix. They provide an efficient implementation of the Procrustes method. The architecture and loss function is the same as for Zhou et al. (2019), except the linear layer produces a 9D vector.
The original work of Liao et al. (2019) only showed results for PASCAL3D+, which used an incorrect evaluation function as noted by Murphy et al. (2021). Thus, for PASCAL3D+, we used the results reported by Murphy et al. ( 2021) with the corrected evaluation function. For ModelNet10-SO(3), we ran their code using a pretrained ResNet50 as an encoder.
We follow the visualization code that was publicly released by Murphy et al. (2021) to represent distributions over 3D rotations as a 2D plot. The probability associated with each rotation is represented as a dot, with the size proportional the magnitude. The rotation is encoded by converting to XYX euler angles, the first two angles correspond to latitude and longitude in a Mollweide projection and the final angle is encoded as color using an HSV colormap. To make the visualizations more interpretable, we do not plot any probabilities less than a given threshold. In our visualizations, we generate probabilities associated with the SO(3) HEALPix grid used during model evaluation (recursion level of 5; 2.4M points).
To efficiently perform SO(3) group convolutions, we must restrict our representations to in the frequency domain. Learning with lower frequency signals can be more efficient and may generalize better in some cases, while higher frequency signals may be better for encoding complex distributions like those shown in Figure 4. In Table 7, we perform a small experiment showing the effects of different maximum frequencies, L, on pose prediction accuracy. Note that we use L = 6 for all other experiments in this work. Interestingly, we find that including higher frequencies in the representation, e.g. L > 6 can actually reduce accuracy, despite the additional learnable parameters. Our method performs one S 2 convolution followed by one SO(3) convolution. In this section, we look at the performance benefits of this final refinement layer and potential benefits of an additional convolution. We find that the performing SO(3) convolution does lead to a marginal improvement, with little to be gained by performing more. This suggests that most of the SO(3) reasoning occurs within the first spherical convolution that uses a learned filter with global support.
See Cohen et al. (2018a) for explanation and visuals describing efficient SO(3) group convolution.
https://e3nn.org
https://github.com/leoshine/Spherical_Regression
https://github.com/google-research/google-research/tree/master/ implicit_pdf
https://cvgl.stanford.edu/projects/pascal3d.html