skip to main content

Title: Generative PointNet: deep energy-based learning on unordered point sets for 3D generation, reconstruction and classification
We propose a generative model of unordered point sets, such as point clouds, in the form of an energy-based model, where the energy function is parameterized by an input permutation- invariant bottom-up neural network. The energy function learns a coordinate encoding of each point and then aggregates all individual point features into an energy for the whole point cloud. We call our model the Generative PointNet because it can be derived from the discriminative PointNet. Our model can be trained by MCMC based maximum likelihood learning (as well as its variants), without the help of any assisting networks like those in GANs and VAEs. Unlike most point cloud generators that rely on hand-crafted distance metrics, our model does not require any hand-crafted distance metric for the point cloud generation, because it synthesizes point clouds by matching observed examples in terms of statistical properties defined by the energy function. Furthermore, we can learn a short run MCMC toward the energy-based model as a flow-like generator for point cloud reconstruction and interpolation. The learned point cloud representation can be useful for point cloud classification. Experiments demonstrate the advantages of the proposed generative model of point clouds.
; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
IEEE Conference on Computer Vision and Pattern Recognition
Sponsoring Org:
National Science Foundation
More Like this
  1. Aims. Thanks to the high angular resolution, sensitivity, image fidelity, and frequency coverage of ALMA, we aim to improve our understanding of star formation. One of the breakthroughs expected from ALMA, which is the basis of our Cycle 5 ALMA-IMF Large Program, is the question of the origin of the initial mass function (IMF) of stars. Here we present the ALMA-IMF protocluster selection, first results, and scientific prospects. Methods. ALMA-IMF imaged a total noncontiguous area of ~53 pc 2 , covering extreme, nearby protoclusters of the Milky Way. We observed 15 massive (2.5 −33 × 10 3 M ⊙ ),more »nearby (2−5.5 kpc) protoclusters that were selected to span relevant early protocluster evolutionary stages. Our 1.3 and 3 mm observations provide continuum images that are homogeneously sensitive to point-like cores with masses of ~0.2 M ⊙ and ~0.6 M ⊙ , respectively, with a matched spatial resolution of ~2000 au across the sample at both wavelengths. Moreover, with the broad spectral coverage provided by ALMA, we detect lines that probe the ionized and molecular gas, as well as complex molecules. Taken together, these data probe the protocluster structure, kinematics, chemistry, and feedback over scales from clouds to filaments to cores. Results. We classify ALMA-IMF protoclusters as Young (six protoclusters), Intermediate (five protoclusters), or Evolved (four proto-clusters) based on the amount of dense gas in the cloud that has potentially been impacted by H  II region(s). The ALMA-IMF catalog contains ~700 cores that span a mass range of ~0.15 M ⊙ to ~250 M ⊙ at a typical size of ~2100 au. We show that this core sample has no significant distance bias and can be used to build core mass functions (CMFs) at similar physical scales. Significant gas motions, which we highlight here in the G353.41 region, are traced down to core scales and can be used to look for inflowing gas streamers and to quantify the impact of the possible associated core mass growth on the shape of the CMF with time. Our first analysis does not reveal any significant evolution of the matter concentration from clouds to cores (i.e., from 1 pc to 0.01 pc scales) or from the youngest to more evolved protoclusters, indicating that cloud dynamical evolution and stellar feedback have for the moment only had a slight effect on the structure of high-density gas in our sample. Furthermore, the first-look analysis of the line richness toward bright cores indicates that the survey encompasses several tens of hot cores, of which we highlight the most massive in the G351.77 cloud. Their homogeneous characterization can be used to constrain the emerging molecular complexity in protostars of high to intermediate masses. Conclusions. The ALMA-IMF Large Program is uniquely designed to transform our understanding of the IMF origin, taking the effects of cloud characteristics and evolution into account. It will provide the community with an unprecedented database with a high legacy value for protocluster clouds, filaments, cores, hot cores, outflows, inflows, and stellar clusters studies.« less
  2. In high energy physics (HEP), jets are collections of correlated particles produced ubiquitously in particle collisions such as those at the CERN Large Hadron Collider (LHC). Machine-learning-based generative models, such as generative adversarial networks (GANs), have the potential to significantly accelerate LHC jet simulations. However, despite jets having a natural representation as a set of particles in momentum-space, a.k.a. a particle cloud, to our knowledge there exist no generative models applied to such a dataset. We introduce a new particle cloud dataset (JetNet), and, due to similarities between particle and point clouds, apply to it existing point cloud GANs. Resultsmore »are evaluated using (1) the 1-Wasserstein distance between high- and low-level feature distributions, (2) a newly developed Fréchet ParticleNet Distance, and (3) the coverage and (4) minimum matching distance metrics. Existing GANs are found to be inadequate for physics applications, hence we develop a new message passing GAN (MPGAN), which outperforms existing point cloud GANs on virtually every metric and shows promise for use in HEP. We propose JetNet as a novel point-cloud-style dataset for the machine learning community to experiment with, and set MPGAN as a benchmark to improve upon for future generative models.« less
  3. In this paper, we examine the long-neglected yet important effects of point sampling patterns in point cloud GANs. Through extensive experiments, we show that sampling-insensitive discriminators (e.g.PointNet-Max) produce shape point clouds with point clustering artifacts while sampling-oversensitive discriminators (e.g. PointNet++, DGCNN) fail to guide valid shape generation. We propose the concept of sampling spectrum to depict the different sampling sensitivities of discriminators. We further study how different evaluation metrics weigh the sampling pattern against the geometry and propose several perceptual metrics forming a sampling spectrum of metrics. Guided by the proposed sampling spectrum, we discover a middle-point sampling-aware baseline discriminator,more »PointNet-Mix, which improves all existing point cloud generators by a large margin on sampling-related metrics. We point out that, though recent research has been focused on the generator design, the main bottleneck of point cloud GAN actually lies in the discriminator design. Our work provides both suggestions and tools for building future discriminators. We will release the code to facilitate future research.« less
  4. ABSTRACT: Molecular simulations with atomistic or coarse- 6 grained force fields are a powerful approach for understanding and 7 predicting the self-assembly phase behavior of complex molecules. 8 Amphiphiles, block oligomers, and block polymers can form 9 mesophases with different ordered morphologies describing the 10 spatial distribution of the blocks, but entirely amorphous nature for 11 local packing and chain conformation. Screening block oligomer 12 chemistry and architecture through molecular simulations to find 13 promising candidates for functional materials is aided by effective 14 and straightforward morphology identification techniques. Captur- 15 ing 3-dimensional periodic structures, such as ordered network 16more »morphologies, is hampered by the requirement that the number of 17 molecules in the simulated system and the shape of the periodic simulation box need to be commensurate with those of the resulting 18 network phase. Common strategies for structure identification include structure factors and order parameters, but these fail to 19 identify imperfect structures in simulations with incorrect system sizes. Building upon pioneering work by DeFever et al. [Chem. Sci. 20 2019, 10, 7503−7515] who implemented a PointNet (i.e., a neural network designed for computer vision applications using point 21 clouds) to detect local structure in simulations of single-bead particles and water molecules, we present a PointNet for detection of 22 nonlocal ordered morphologies of complex block oligomers. Our PointNet was trained using atomic coordinates from molecular 23 dynamics simulation trajectories and synthetic point clouds for ordered network morphologies that were absent from previous 24 simulations. In contrast to prior work on simple molecules, we observe that large point clouds with 1000 or more points are needed 25 for the more complex block oligomers. The trained PointNet model achieves an accuracy as high as 0.99 for globally ordered 26 morphologies formed by linear diblock, linear triblock, and 3-arm and 4-arm star-block oligomers, and it also allows for the discovery 27 of emerging ordered patterns from nonequilibrium systems.« less
  5. We propose CaSPR, a method to learn object-centric Canonical Spatiotemporal Point Cloud Representations of dynamically moving or evolving objects. Our goal is to enable information aggregation over time and the interrogation of object state at any spatiotemporal neighborhood in the past, observed or not. Different from previous work, CaSPR learns representations that support spacetime continuity, are robust to variable and irregularly spacetime-sampled point clouds, and generalize to unseen object instances. Our approach divides the problem into two subtasks. First, we explicitly encode time by mapping an input point cloud sequence to a spatiotemporally-canonicalized object space. We then leverage this canonicalizationmore »to learn a spatiotemporal latent representation using neural ordinary differential equations and a generative model of dynamically evolving shapes using continuous normalizing flows. We demonstrate the effectiveness of our method on several applications including shape reconstruction, camera pose estimation, continuous spatiotemporal sequence reconstruction, and correspondence estimation from irregularly or intermittently sampled observations.« less