This paper introduces a deep neural network based method, i.e., DeepOrganNet, to generate and visualize fully high-fidelity 3D / 4D organ geometric models from single-view medical images with complicated background in real time. Traditional 3D / 4D medical image reconstruction requires near hundreds of projections, which cost insufferable computational time and deliver undesirable high imaging / radiation dose to human subjects. Moreover, it always needs further notorious processes to segment or extract the accurate 3D organ models subsequently. The computational time and imaging dose can be reduced by decreasing the number of projections, but the reconstructed image quality is degraded accordingly. To our knowledge, there is no method directly and explicitly reconstructing multiple 3D organ meshes from a single 2D medical grayscale image on the fly. Given single-view 2D medical images, e.g., 3D / 4D-CT projections or X-ray images, our end-to-end DeepOrganNet framework can efficiently and effectively reconstruct 3D / 4D lung models with a variety of geometric shapes by learning the smooth deformation fields from multiple templates based on a trivariate tensor-product deformation technique, leveraging an informative latent descriptor extracted from input 2D images. The proposed method can guarantee to generate high-quality and high-fidelity manifold meshes for 3D /more »
Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
Monocular 3D object parsing is highly desirable in various scenarios including occlusion reasoning and holistic
scene interpretation. We present a deep convolutional neural network (CNN) architecture to localize semantic
parts in 2D image and 3D space while inferring their visibility states, given a single RGB image. Our key insight
is to exploit domain knowledge to regularize the network by deeply supervising its hidden layers, in order to
sequentially infer intermediate concepts associated with the final task. To acquire training data in desired
quantities with ground truth 3D shape and relevant concepts, we render 3D object CAD models to generate
large-scale synthetic data and simulate challenging occlusion configurations between objects. We train the
network only on synthetic data and demonstrate state-of-the-art performances on real image benchmarks
including an extended version of KITTI, PASCAL VOC, PASCAL3D+ and IKEA for 2D and 3D keypoint
localization and instance segmentation. The empirical results substantiate the utility of our deep supervision
scheme by demonstrating effective transfer of knowledge from synthetic data to real images, resulting in less
overfitting compared to standard end-to-end training.
- Award ID(s):
- 1637949
- Publication Date:
- NSF-PAR ID:
- 10047463
- Journal Name:
- CVPR
- Page Range or eLocation-ID:
- 388 to 397
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
We introduce an interactive Soft Shadow Network (SSN) to generates controllable soft shadows for image composit- ing. SSN takes a 2D object mask as input and thus is ag- nostic to image types such as painting and vector art. An environment light map is used to control the shadow’s char- acteristics, such as angle and softness. SSN employs an Ambient Occlusion Prediction module to predict an inter- mediate ambient occlusion map, which can be further re- fined by the user to provides geometric cues to modulate the shadow generation. To train our model, we design an efficient pipeline to produce diverse soft shadow training data using 3D object models. In addition, we propose an inverse shadow map representation to improve model train- ing. We demonstrate that our model produces realistic soft shadows in real-time. Our user studies show that the gen- erated shadows are often indistinguishable from shadows calculated by a physics-based renderer and users can eas- ily use SSN through an interactive application to generate specific shadow effects in minutes.
-
We present a new weakly supervised learning-based method for generating novel category-specific 3D shapes from unoccluded image collections. Our method is weakly supervised and only requires silhouette annotations from unoccluded, category-specific objects. Our method does not require access to the object's 3D shape, multiple observations per object from different views, intra-image pixel correspondences, or any view annotations. Key to our method is a novel multi-projection generative adversarial network (MP-GAN) that trains a 3D shape generator to be consistent with multiple 2D projections of the 3D shapes, and without direct access to these 3D shapes. This is achieved through multiple discriminators that encode the distribution of 2D projections of the 3D shapes seen from a different views. Additionally, to determine the view information for each silhouette image, we also train a view prediction network on visualizations of 3D shapes synthesized by the generator. We iteratively alternate between training the generator and training the view prediction network. We validate our multi-projection GAN on both synthetic and real image datasets. Furthermore, we also show that multi-projection GANs can aid in learning other high-dimensional distributions from lower dimensional training datasets, such as material-class specific spatially varying reflectance properties from images.
-
Medical image segmentation is one of the most challenging tasks in medical image analysis and widely developed for many clinical applications. While deep learning-based approaches have achieved impressive performance in semantic segmentation, they are limited to pixel-wise settings with imbalanced-class data problems and weak boundary object segmentation in medical images. In this paper, we tackle those limitations by developing a new two-branch deep network architecture which takes both higher level features and lower level features into account. The first branch extracts higher level feature as region information by a common encoder-decoder network structure such as Unet and FCN, whereas the second branch focuses on lower level features as support information around the boundary and processes in parallel to the first branch. Our key contribution is the second branch named Narrow Band Active Contour (NB-AC) attention model which treats the object contour as a hyperplane and all data inside a narrow band as support information that influences the position and orientation of the hyperplane. Our proposed NB-AC attention model incorporates the contour length with the region energy involving a fixed-width band around the curve or surface. The proposed network loss contains two fitting terms: (i) a high level feature (i.e., region)more »
-
Material and biological sciences frequently generate large amounts of microscope data that require 3D object level segmentation. Often, the objects of interest have a common geometry, for example spherical, ellipsoidal, or cylindrical shapes. Neural networks have became a popular approach for object detection but they are often limited by their training dataset and have difficulties adapting to new data. In this paper, we propose a volumetric object detection approach for microscopy volumes comprised of fibrous structures by using deep centroid regression and geometric regularization. To this end, we train encoder-decoder networks for segmentation and centroid regression. We use the regression information combined with prior system knowledge to propose cylindrical objects and enforce geometric regularization in the segmentation. We train our networks on synthetic data and then test the trained networks in several experimental datasets. Our approach shows competitive results against other 3D segmentation methods when tested on the synthetic data and outperforms those other methods across different datasets.
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Publisher's Version of Record
- https://doi.org/10.1109/CVPR.2017.49
