skip to main content


Title: : Joint Point Interaction-Dimension Search for 3D Point Cloud
The interaction and dimension of points are two important axes in designing point operators to serve hierarchical 3D models. Yet, these two axes are heterogeneous and challenging to fully explore. Existing works craft point operator under a single axis and reuse the crafted operator in all parts of 3D models. This overlooks the opportunity to better combine point interactions and dimensions by exploiting varying geometry/density of 3D point clouds. In this work, we establish PIDS, a novel paradigm to jointly explore point interactions and point dimensions to serve semantic segmentation on point cloud data. We establish a large search space to jointly consider versatile point interactions and point dimensions. This supports point operators with various geometry/density considerations. The enlarged search space with heterogeneous search components calls for a better ranking of candidate models. To achieve this, we improve the search space exploration by leveraging predictor-based Neural Architecture Search (NAS), and enhance the quality of prediction by assigning unique encoding to heterogeneous search components based on their priors. We thoroughly evaluate the networks crafted by PIDS on two semantic segmentation benchmarks, showing ∼ 1% mIOU improvement on SemanticKITTI and S3DIS over state-of-the-art 3D models.  more » « less
Award ID(s):
1937435 2112562 2048044
NSF-PAR ID:
10396503
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)
Page Range / eLocation ID:
1298 to 1307
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. The interaction and dimension of points are two important axes in designing point operators to serve hierarchical 3D models. Yet, these two axes are heterogeneous and challenging to fully explore. Existing works craft point operator under a single axis and reuse the crafted operator in all parts of 3D models. This overlooks the opportunity to better combine point interactions and dimensions by exploiting varying geometry/density of 3D point clouds. In this work, we establish PIDS, a novel paradigm to jointly explore point interactions and point dimensions to serve semantic segmentation on point cloud data. We establish a large search space to jointly consider versatile point interactions and point dimensions. This supports point operators with various geometry/density considerations. The enlarged search space with heterogeneous search components calls for a better ranking of candidate models. To achieve this, we improve the search space exploration by leveraging predictor-based Neural Architecture Search (NAS), and enhance the quality of prediction by assigning unique encoding to heterogeneous search components based on their priors. We thoroughly evaluate the networks crafted by PIDS on two semantic segmentation benchmarks, showing 1% mIOU improvement on SemanticKITTI and S3DIS over state-of-the-art 3D models. 
    more » « less
  2. Analyzing the geometric and semantic properties of 3D point clouds through the deep networks is still challenging due to the irregularity and sparsity of samplings of their geometric structures. This paper presents a new method to define and compute convolution directly on 3D point clouds by the proposed annular convolution. This new convolution operator can better capture the local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processing level. We apply it to the developed hierarchical neural networks for object classification, part segmentation, and semantic segmentation in large-scale scenes. The extensive experiments and comparisons demonstrate that our approach outperforms the state-of-the-art methods on a variety of standard benchmark datasets (e.g., ModelNet10, ModelNet40, ShapeNetpart, S3DIS, and ScanNet). 
    more » « less
  3. Analyzing the geometric and semantic properties of 3D point clouds through the deep networks is still challenging due to the irregularity and sparsity of samplings of their geometric structures. This paper presents a new method to define and compute convolution directly on 3D point clouds by the proposed annular convolution. This new convolution operator can better capture the local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processing level. We apply it to the developed hierarchical neural networks for object classification, part segmentation, and semantic segmentation in large-scale scenes. The extensive experiments and comparisons demonstrate that our approach outperforms the state-of-the-art methods on a variety of standard benchmark datasets (e.g., ModelNet10, ModelNet40, ShapeNetpart, S3DIS, and ScanNet). 
    more » « less
  4. Unlike images which are represented in regular dense grids, 3D point clouds are irregular and unordered, hence applying convolution on them can be difficult. In this paper, we extend the dynamic filter to a new convolution operation, named PointConv. PointConv can be applied on point clouds to build deep convolutional networks. We treat convolution kernels as nonlinear functions of the local coordinates of 3D points comprised of weight and density functions. With respect to a given point, the weight functions are learned with multi-layer perceptron networks and the density functions through kernel density estimation. A novel reformulation is proposed for efficiently computing the weight functions, which allowed us to dramatically scale up the network and significantly improve its performance. The learned convolution kernel can be used to compute translation-invariant and permutation-invariant convolution on any point set in the 3D space. Besides, PointConv can also be used as deconvolution operators to propagate features from a subsampled point cloud back to its original resolution. Experiments on ModelNet40, ShapeNet, and ScanNet show that deep convolutional neural networks built on PointConv are able to achieve state-of-the-art on challenging semantic segmentation benchmarks on 3D point clouds. Besides, our experiments converting CIFAR-10 into a point cloud showed that networks built on PointConv can match the performance of convolutional networks in 2D images of a similar structure. 
    more » « less
  5. A bstract We study monodromy defects in O ( N ) symmetric scalar field theories in d dimensions. After a Weyl transformation, a monodromy defect may be described by placing the theory on S 1 × H d− 1 , where H d− 1 is the hyperbolic space, and imposing on the fundamental fields a twisted periodicity condition along S 1 . In this description, the codimension two defect lies at the boundary of H d− 1 . We first study the general monodromy defect in the free field theory, and then develop the large N expansion of the defect in the interacting theory, focusing for simplicity on the case of N complex fields with a one-parameter monodromy condition. We also use the ϵ -expansion in d = 4 − ϵ , providing a check on the large N approach. When the defect has spherical geometry, its expectation value is a meaningful quantity, and it may be obtained by computing the free energy of the twisted theory on S 1 × H d− 1 . It was conjectured that the logarithm of the defect expectation value, suitably multiplied by a dimension dependent sine factor, should decrease under a defect RG flow. We check this conjecture in our examples, both in the free and interacting case, by considering a defect RG flow that corresponds to imposing alternate boundary conditions on one of the low-lying Kaluza-Klein modes on H d− 1 . We also show that, adapting standard techniques from the AdS/CFT literature, the S 1 × H d− 1 setup is well suited to the calculation of the defect CFT data, and we discuss various examples, including one-point functions of bulk operators, scaling dimensions of defect operators, and four-point functions of operator insertions on the defect. 
    more » « less