In this paper, we present a novel Deformable Neural Articulations Network (DNA-Net), which is a template- free learning-based method for dynamic 3D human reconstruction from a single RGB-D sequence. Our proposed DNA-Net includes a Neural Articulation Prediction Net- work (NAP-Net), which is capable of representing non-rigid motions of a human by learning to predict a set of articulated bones to follow movements of the human in the in- put sequence. Moreover, DNA-Net also include Signed Distance Field Network (SDF-Net) and Appearance Network (Color-Net), which take advantage of the powerful neural implicit functions in modeling 3D geometries and appear- ance. Finally, to avoid the reliance on external optical flow estimators to obtain deformation cues like previous related works, we propose a novel training loss, namely Easy-to- Hard Geometric-based, which is a simple strategy that inherits the merits of Chamfer distance to achieve good de- formation guidance while still avoiding its limitation of lo- cal mismatches sensitivity. DNA-Net is trained end-to-end in a self-supervised manner directly on the input sequence to obtain 3D reconstructions of the input objects. Quantitative results on videos of DeepDeform dataset show that DNA-Net outperforms related state-of-the-art methods with an adequate gaps, qualitative results additionally prove that our method can reconstruct human shapes with high fidelity and details.
more »
« less
Topology-Aware Single-Image 3D Shape Reconstruction
We make an attempt to address topology-awareness for 3D shape reconstruction. Two types of high-level shape typologies are being studied here, namely genus (number of cuttings/holes) and connectivity (number of connected components), which are of great importance in 3D object reconstruction/understanding but have been thus far disjoint from the existing dense voxel-wise prediction literature. We propose a topology-aware shape autoencoder component (TPWCoder) by approximating topology property functions such as genus and connectivity with neural networks from the latent variables. TPWCoder can be directly combined with the existing 3D shape reconstruction pipelines for end-to-end training and prediction. On the challenging A Big CAD Model Dataset (ABC), TPWCoder demonstrates a noticeable quantitative and qualitative improvement over the competing methods, and it also shows improved quantitative result on the ShapeNet dataset.
more »
« less
- NSF-PAR ID:
- 10166846
- Date Published:
- Journal Name:
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition workshops
- ISSN:
- 2160-7516
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
more » « less
Quantitative volumetric assessment of filamentous actin (F‐actin) fibers remains challenging due to their interconnected nature, leading researchers to utilize threshold‐based or qualitative measurement methods with poor reproducibility. Herein, a novel machine learning‐based methodology is introduced for accurate quantification and reconstruction of nuclei‐associated F‐actin. Utilizing a convolutional neural network (CNN), actin filaments and nuclei from 3D confocal microscopy images are segmented and then each fiber is reconstructed by connecting intersecting contours on cross‐sectional slices. This allows measurement of the total number of actin filaments and individual actin filament length and volume in a reproducible fashion. Focusing on the role of F‐actin in supporting nucleocytoskeletal connectivity, apical F‐actin, basal F‐actin, and nuclear architecture in mesenchymal stem cells (MSCs) are quantified following the disruption of the linker of nucleoskeleton and cytoskeleton (LINC) complexes. Disabling LINC in MSCs generates F‐actin disorganization at the nuclear envelope characterized by shorter length and volume of actin fibers contributing a less elongated nuclear shape. The findings not only present a new tool for mechanobiology but introduce a novel pipeline for developing realistic computational models based on quantitative measures of F‐actin.
-
Capturing document images with hand-held devices in unstructured environments is a common practice nowadays. However, “casual” photos of documents are usually unsuitable for automatic information extraction, mainly due to physical distortion of the document paper, as well as various camera positions and illumination conditions. In this work, we propose DewarpNet, a deep-learning approach for document image unwarping from a single image. Our insight is that the 3D geometry of the document not only determines the warping of its texture but also causes the illumination effects. Therefore, our novelty resides on the explicit modeling of 3D shape for document paper in an end-to-end pipeline. Also, we contribute the largest and most comprehensive dataset for document image unwarping to date – Doc3D. This dataset features multiple ground-truth annotations, including 3D shape, surface normals, UV map, albedo image, etc. Training with Doc3D, we demonstrate state-of-the-art performance for DewarpNet with extensive qualitative and quantitative evaluations. Our network also significantly improves OCR performance on captured document images, decreasing character error rate by 42% on average. Both the code and the dataset are released.more » « less
-
Dynamic network topology can pose important challenges to communication and control protocols in networks of autonomous vehicles. For instance, maintaining connectivity is a key challenge in unmanned aerial vehicle (UAV) networks. However, tracking and computational resources of the observer module might not be sufficient for constant monitoring of all surrounding nodes in large-scale networks. In this paper, we propose an optimal measurement policy for network topology monitoring under constrained resources. To this end, We formulate the localization of multiple objects in terms of linear networked systems and solve it using Kalman filtering with intermittent observation. The proposed policy includes two sequential steps. We first find optimal measurement attempt probabilities for each target using numerical optimization methods to assign the limited number of resources among targets. The optimal resource allocation follows a waterfall-like solution to assign more resources to targets with lower measurement success probability. This provides a 10% to 60% gain in prediction accuracy. The second step is finding optimal on-off patterns for measurement attempts for each target over time. We show that a regular measurement pattern that evenly distributed resources over time outperforms the two extreme cases of using all measurement resources either in the beginning or at the end of the measurement cycle. Our proof is based on characterizing the fixed-point solution of the error covariance matrix for regular patterns. Extensive simulation results confirm the optimality of the most alternating pattern with up to 10-fold prediction improvement for different scenarios. These two guidelines define a general policy for target tracking under constrained resources with applications to network topology prediction of autonomous systemsmore » « less
-
null (Ed.)We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views. Previous work on learning shape reconstruction from multiple views uses discrete representations such as point clouds or voxels, while continuous surface generation approaches lack multi-view consistency. We address these issues by designing neural networks capable of generating high-quality parametric 3D surfaces which are also consistent between views. Furthermore, the generated 3D surfaces preserve accurate image pixel to 3D surface point correspondences, allowing us to lift texture information to reconstruct shapes with rich geometry and appearance. Our method is supervised and trained on a public dataset of shapes from common object categories. Quantitative results indicate that our method significantly outperforms previous work, while qualitative results demonstrate the high quality of our reconstructions.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
