skip to main content

Title: IMEXnet - A Forward Stable Deep Neural Network
Deep convolutional neural networks have revolutionized many machine learning and computer vision tasks, however, some remaining key challenges limit their wider use. These challenges include improving the network's robustness to perturbations of the input image and the limited ``field of view'' of convolution operators. We introduce the IMEXnet that addresses these challenges by adapting semi-implicit methods for partial differential equations. Compared to similar explicit networks, such as residual networks, our network is more stable, which has recently shown to reduce the sensitivity to small changes in the input features and improve generalization. The addition of an implicit step connects all pixels in each channel of the image and therefore addresses the field of view problem while still being comparable to standard convolutions in terms of the number of parameters and computational complexity. We also present a new dataset for semantic segmentation and demonstrate the effectiveness of our architecture using the NYU Depth dataset.
; ; ;
Award ID(s):
Publication Date:
Journal Name:
International Conference on Machine Learning
Sponsoring Org:
National Science Foundation
More Like this
  1. We present EgoRenderer, a system for rendering full-body neural avatars of a person captured by a wearable, egocentric fisheye camera that is mounted on a cap or a VR headset. Our system renders photorealistic novel views of the actor and her motion from arbitrary virtual camera locations. Rendering full-body avatars from such egocentric images come with unique challenges due to the top-down view and large distortions. We tackle these challenges by decomposing the rendering process into several steps, including texture synthesis, pose construction, and neural image translation. For texture synthesis, we propose Ego-DPNet, a neural network that infers dense correspondences between the input fisheye images and an underlying parametric body model, and to extract textures from egocentric inputs. In addition, to encode dynamic appearances, our approach also learns an implicit texture stack that captures detailed appearance variation across poses and viewpoints. For correct pose generation, we first estimate body pose from the egocentric view using a parametric model. We then synthesize an external free-viewpoint pose image by projecting the parametric model to the user-specified target viewpoint. We next combine the target pose image and the textures into a combined feature image, which is transformed into the output color image usingmore »a neural image translation network. Experimental evaluations show that EgoRenderer is capable of generating realistic free-viewpoint avatars of a person wearing an egocentric camera. Comparisons to several baselines demonstrate the advantages of our approach.« less
  2. In accelerated MRI reconstruction, the anatomy of a patient is recovered from a set of under-sampled and noisy measurements. Deep learning approaches have been proven to be successful in solving this ill-posed inverse problem and are capable of producing very high quality reconstructions. However, current architectures heavily rely on convolutions, that are content-independent and have difficulties modeling long-range dependencies in images. Recently, Transformers, the workhorse of contemporary natural language processing, have emerged as powerful building blocks for a multitude of vision tasks. These models split input images into nonoverlapping patches, embed the patches into lower-dimensional tokens and utilize a self-attention mechanism that does not suffer from the aforementioned weaknesses of convolutional architectures. However, Transformers incur extremely high compute and memory cost when 1) the input image resolution is high and 2) when the image needs to be split into a large number of patches to preserve fine detail information, both of which are typical in low-level vision problems such as MRI reconstruction, having a compounding effect. To tackle these challenges, we propose HUMUS-Net, a hybrid architecture that combines the beneficial implicit bias and efficiency of convolutions with the power of Transformer blocks in an unrolled and multi-scale network. HUMUS-Net extractsmore »high-resolution features via convolutional blocks and refines low-resolution features via a novel Transformer-based multi-scale feature extractor. Features from both levels are then synthesized into a high-resolution output reconstruction. Our network establishes new state of the art on the largest publicly available MRI dataset, the fastMRI dataset. We further demonstrate the performance of HUMUS-Net on two other popular MRI datasets and perform fine-grained ablation studies to validate our design.« less
  3. Vitreous collagen structure plays an important role in ocular mechanics. However, capturing this structure with existing vitreous imaging methods is hindered by the loss of sample position and orientation, low resolution, or a small field of view. The objective of this study was to evaluate confocal reflectance microscopy as a solution to these limitations. Intrinsic reflectance avoids staining, and optical sectioning eliminates the requirement for thin sectioning, minimizing processing for optimal preservation of the natural structure. We developed a sample preparation and imaging strategy usingex vivogrossly sectioned porcine eyes. Imaging revealed a network of uniform diameter crossing fibers (1.1 ± 0.3 µm for a typical image) with generally poor alignment (alignment coefficient = 0.40 ± 0.21 for a typical image). To test the utility of our approach for detecting differences in fiber spatial distribution, we imaged eyes every 1 mm along an anterior-posterior axis originating at the limbus and quantified the number of fibers in each image. Fiber density was higher anteriorly near the vitreous base, regardless of the imaging plane. These data demonstrate that confocal reflectance microscopy addresses the previously unmet need for a robust, micron-scale technique to map features of collagen networksin situacross the vitreous.

  4. Implicit neural networks are a general class of learning models that replace the layers in traditional feedforward models with implicit algebraic equations. Compared to traditional learning models, implicit networks offer competitive performance and reduced memory consumption. However, they can remain brittle with respect to input adversarial perturbations. This paper proposes a theoretical and computational framework for robustness verification of implicit neural networks; our framework blends together mixed monotone systems theory and contraction theory. First, given an implicit neural network, we introduce a related embedded network and show that, given an infinity-norm box constraint on the input, the embedded network provides an infinity-norm box overapproximation for the output of the original network. Second, using infinity-matrix measures, we propose sufficient conditions for well-posedness of both the original and embedded system and design an iterative algorithm to compute the infinity-norm box robustness margins for reachability and classification problems. Third, of independent value, we show that employing a suitable relative classifier variable in our analysis will lead to tighter bounds on the certified adversarial robustness in classification problems. Finally, we perform numerical simulations on a Non-Euclidean Monotone Operator Network (NEMON) trained on the MNIST dataset. In these simulations, we compare the accuracy and runmore »time of our mixed monotone contractive approach with the existing robustness verification approaches in the literature for estimating the certified adversarial robustness.« less
  5. Traditional engineering courses typically approach teaching and problem solving by focusing on the physical dimensions of those problems without consideration of dynamic social and ethical dimensions. As such, projects can fail to consider community questions and concerns, broader impacts upon society, or otherwise result in inequitable outcomes. And, despite the fact that students in engineering receive training on the Professional Code of Ethics for Engineers, to which they are expected to adhere in practice, many students are unable to recognize and analyze real-life ethical challenges as they arise. Indeed, research has found that students are typically less engaged with ethics—defined as the awareness and judgment of microethics and macroethics, sensitivity to diversity, and interest in promoting organizational ethical culture—at the end of their engineering studies than they were at the beginning. As such, many studies have focused on developing and improving the curriculum surrounding ethics through, for instance, exposing students to ethics case studies. However, such ethics courses often present a narrow and simplified view of ethics that students may struggle to integrate with their broader experience as engineers. Thus, there is a critical need to unpack the complexity of ethical behavior amongst engineering students in order to determine howmore »to better foster ethical judgment and behavior. Promoting ethical behavior among engineering students and developing a culture of ethical behavior within institutions have become goals of many engineering programs. Towards this goal, we present an overview of the current scholarship of engineering ethics and propose a theoretical framework of ethical behavior using a review of articles related to engineering ethics from 1990-2020. These articles were selected based upon their diversity of scope and methods until saturation was reached. A thematic analysis of articles was then performed using Nvivo. The review engages in theories across disciplines including philosophy, education and psychology. Preliminary results identify two major kinds of drivers of ethical behavior, namely individual level ethical behavior drivers (awareness of microethics, awareness of macroethics, implicit understanding, and explicit understanding) and institutional drivers (diversity and institutional ethical culture). In this paper, we present an overview and discussion of two drivers of ethical behavior at the individual level, namely awareness of microethics and awareness of macroethics, based on a review of 50 articles. Our results indicate that an awareness of both microethics and macroethics is essential in promoting ethical behavior amongst students. The review also points to a need to focus on increasing students’ awareness of macroethics. This research thus addresses the need, driven by existing scholarship, to identify a conceptual framework for explaining how ethical judgment and behavior in engineering can be further promoted.« less