We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera that is mounted on a cap or a VR headset. Our system renders photorealistic novel views of the actor and her motion from arbitrary virtual camera locations. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation. For texture synthesis, we propose Ego-DPNet, a neural network that infers dense correspondences between the input fisheye images and an underlying parametric body model, and to extract textures from egocentric inputs. In addition, to encode dynamic appearances, our approach also learns an implicit texture stack that captures detailed appearance variation across poses and viewpoints. For correct pose generation, we first estimate body pose from the egocentric view using a parametric model. We then synthesize an external free-viewpoint pose image by projecting the parametric model to the user-specified target viewpoint. We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image usingmore »
IMEXnet - A Forward Stable Deep Neural Network
Deep convolutional neural networks have revolutionized many machine learning and computer vision tasks, however, some remaining key challenges limit their wider use. These challenges include improving the network's robustness to perturbations of the input image and the limited ``field of view'' of convolution operators. We introduce the IMEXnet that addresses these challenges by adapting semi-implicit methods for partial differential equations. Compared to similar explicit networks, such as residual networks, our network is more stable, which has recently shown to reduce the sensitivity to small changes in the input features and improve generalization. The addition of an implicit step connects all pixels in each channel of the image and therefore addresses the field of view problem while still being comparable to standard convolutions in terms of the number of parameters and computational complexity. We also present a new dataset for semantic segmentation and demonstrate the effectiveness of our architecture using the NYU Depth dataset.
- Award ID(s):
- 1751636
- Publication Date:
- NSF-PAR ID:
- 10095794
- Journal Name:
- International Conference on Machine Learning
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements. Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely on convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks. These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures. However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve fine detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect. To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit bias and efficiency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network. HUMUS-Net extractsmore »
-
Vitreous collagen structure plays an important role in ocular mechanics. However, capturing this structure with existing vitreous imaging methods is hindered by the loss of sample position and orientation, low resolution, or a small field of view. The objective of this study was to evaluate confocal reflectance microscopy as a solution to these limitations. Intrinsic reflectance avoids staining, and optical sectioning eliminates the requirement for thin sectioning, minimizing processing for optimal preservation of the natural structure. We developed a sample preparation and imaging strategy using
ex vivo grossly sectioned porcine eyes. Imaging revealed a network of uniform diameter crossing fibers (1.1 ± 0.3 µm for a typical image) with generally poor alignment (alignment coefficient = 0.40 ± 0.21 for a typical image). To test the utility of our approach for detecting differences in fiber spatial distribution, we imaged eyes every 1 mm along an anterior-posterior axis originating at the limbus and quantified the number of fibers in each image. Fiber density was higher anteriorly near the vitreous base, regardless of the imaging plane. These data demonstrate that confocal reflectance microscopy addresses the previously unmet need for a robust, micron-scale technique to map features of collagen networksin situ across the vitreous. -
Implicit neural networks are a general class of learning models that replace the layers in traditional feedforward models with implicit algebraic equations. Compared to traditional learning models, implicit networks offer competitive performance and reduced memory consumption. However, they can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks; our framework blends together mixed monotone systems theory and contraction theory. First, given an implicit neural network, we introduce a related embedded network and show that, given an infinity-norm box constraint on the input, the embedded network provides an infinity-norm box overapproximation for the output of the original network. Second, using infinity-matrix measures, we propose sufficient conditions for well-posedness of both the original and embedded system and design an iterative algorithm to compute the infinity-norm box robustness margins for reachability and classification problems. Third, of independent value, we show that employing a suitable relative classifier variable in our analysis will lead to tighter bounds on the certified adversarial robustness in classification problems. Finally, we perform numerical simulations on a Non-Euclidean Monotone Operator Network (NEMON) trained on the MNIST dataset. In these simulations, we compare the accuracy and runmore »
-
Traditional engineering courses typically approach teaching and problem solving by focusing on the physical dimensions of those problems without consideration of dynamic social and ethical dimensions. As such, projects can fail to consider community questions and concerns, broader impacts upon society, or otherwise result in inequitable outcomes. And, despite the fact that students in engineering receive training on the Professional Code of Ethics for Engineers, to which they are expected to adhere in practice, many students are unable to recognize and analyze real-life ethical challenges as they arise. Indeed, research has found that students are typically less engaged with ethics—defined as the awareness and judgment of microethics and macroethics, sensitivity to diversity, and interest in promoting organizational ethical culture—at the end of their engineering studies than they were at the beginning. As such, many studies have focused on developing and improving the curriculum surrounding ethics through, for instance, exposing students to ethics case studies. However, such ethics courses often present a narrow and simplified view of ethics that students may struggle to integrate with their broader experience as engineers. Thus, there is a critical need to unpack the complexity of ethical behavior amongst engineering students in order to determine howmore »
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Publisher's Version of Record
- The DOI is not currently available
