Existing generative models for 3D shapes are typically trained on a large 3D dataset, often of a specific object category. In this paper, we investigate the deep generative model that learns from only a single reference 3D shape. Specifically, we present a multi-scale GAN-based model designed to capture the input shape's geometric features across a range of spatial scales. To avoid large memory and computational cost induced by operating on the 3D volume, we build our generator atop the tri-plane hybrid representation, which requires only 2D convolutions. We train our generative model on a voxel pyramid of the reference shape, without the need of any external supervision or manual annotation. Once trained, our model can generate diverse and high-quality 3D shapes possibly of different sizes and aspect ratios. The resulting shapes present variations across different scales, and at the same time retain the global structure of the reference shape. Through extensive evaluation, both qualitative and quantitative, we demonstrate that our model can generate 3D shapes of various types. 1
more »
« less
LSD-StructureNet: Modeling Levels of Structural Detail in 3D Part Hierarchies
Generative models for 3D shapes represented by hierar- chies of parts can generate realistic and diverse sets of out- puts. However, existing models suffer from the key practi- cal limitation of modelling shapes holistically and thus can- not perform conditional sampling, i.e. they are not able to generate variants on individual parts of generated shapes without modifying the rest of the shape. This is limiting for applications such as 3D CAD design that involve adjust- ing created shapes at multiple levels of detail. To address this, we introduce LSD-StructureNet, an augmentation to the StructureNet architecture that enables re-generation of parts situated at arbitrary positions in the hierarchies of its outputs. We achieve this by learning individual, probabilis- tic conditional decoders for each hierarchy depth. We eval- uate LSD-StructureNet on the PartNet dataset, the largest dataset of 3D shapes represented by hierarchies of parts. Our results show that contrarily to existing methods, LSD- StructureNet can perform conditional sampling without im- pacting inference speed or the realism and diversity of its outputs.
more »
« less
- Award ID(s):
- 2020227
- NSF-PAR ID:
- 10294602
- Date Published:
- Journal Name:
- Int. Conf. Computer Vision
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Synopsis Fish perform many complex manipulation behaviors without hands or flexible muscular tongues, instead relying on more than 20 movable skeletal elements in their highly kinetic skulls. How fish use their skulls to accomplish these behaviors, however, remains unclear. Most previous mechanical models have represented the fish skull using one or more planar four-bar linkages, which have just a single degree of freedom (DoF). In contrast, truncated-cone hydrodynamic models have assumed up to five DoFs. In this study, we introduce and validate a 3D mechanical linkage model of a fish skull that incorporates the pectoral girdle and mandibular and hyoid arches. We validate this model using an in vivo motion dataset of suction feeding in channel catfish and then use this model to quantify the DoFs in the fish skull, to categorize the motion patterns of the cranial linkage during feeding, and to evaluate the association between these patterns and food motion. We find that the channel catfish skull functions as a 17-link, five-loop parallel mechanism. Despite having 19 potential DoFs, we find that seven DoFs are sufficient to describe most of the motion of the cranial linkage, consistent with the fish skull functioning as a multi-DoF, manipulation system. Channel catfish use this linkage to generate three different motion patterns (rostrocaudal wave, caudorostral wave, and compressive wave), each with its own associated food velocity profile. These results suggest that biomechanical manipulation systems must have a minimum number of DoFs to effectively control objects, whether in water or air.more » « less
-
more » « less
Abstract We present a method that detects boundaries of parts in 3D shapes represented as point clouds. Our method is based on a graph convolutional network architecture that outputs a probability for a point to lie in an area that separates two or more parts in a 3D shape. Our boundary detector is quite generic: it can be trained to localize boundaries of semantic parts or geometric primitives commonly used in 3D modeling. Our experiments demonstrate that our method can extract more accurate boundaries that are closer to ground‐truth ones compared to alternatives. We also demonstrate an application of our network to fine‐grained semantic shape segmentation, where we also show improvements in terms of part labeling performance.
-
Manipulating an articulated object requires perceiving its kinematic hierarchy: its parts, how each can move, and how those motions are coupled. Previous work has explored perception for kinematics, but none infers a complete kinematic hierarchy on never-before-seen object instances, without relying on a schema or template. We present a novel perception system that achieves this goal. Our system infers the moving parts of an object and the kinematic couplings that relate them. To infer parts, it uses a point cloud instance segmentation neural network and to infer kinematic hierarchies, it uses a graph neural network to predict the existence, direction, and type of edges (i.e. joints) that relate the inferred parts. We train these networks using simulated scans of synthetic 3D models. We evaluate our system on simulated scans of 3D objects, and we demonstrate a proof-of-concept use of our system to drive real-world robotic manipulation.more » « less
-
more » « less
It is well-known that morphological features in the brain undergo changes due to traumatic events and associated disorders such as post-traumatic stress disorder (PTSD). However, existing approaches typically offer group-level comparisons, and there are limited predictive approaches for modeling behavioral outcomes based on brain shape features that can account for heterogeneity in PTSD, which is of paramount interest. We propose a comprehensive shape analysis framework representing brain sub-structures, such as the hippocampus, amygdala, and putamen, as parameterized surfaces and quantifying their shape differences using an elastic shape metric. Under this metric, we compute shape summaries (mean, covariance, PCA) of brain sub-structures and represent individual brain shapes by their principal scores under a shape-PCA basis. These representations are rich enough to allow visualizations of full 3D structures and help understand localized changes. In order to validate the elastic shape analysis, we use the principal components (PCs) to reconstruct the brain structures and perform further evaluation by performing a regression analysis to model PTSD and trauma severity using the brain shapes represented
via PCs and in conjunction with auxiliary exposure variables. We apply our method to data from the Grady Trauma Project (GTP), where the goal is to predict clinical measures of PTSD. The framework seamlessly integrates accurate morphological features and other clinical covariates to yield superior predictive performance when modeling PTSD outcomes. Compared to vertex-wise analysis and other widely applied shape analysis methods, the elastic shape analysis approach results in considerably higher reconstruction accuracy for the brain shape and reveals significantly greater predictive power. It also helps identify local deformations in brain shapes associated with PTSD severity.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
