An official website of the United States government
This paper explores the synergistic potential of neuromorphic and edge computing to create a versatile machine learning (ML) system tailored for processing data captured by dynamic vision sensors. We construct and train hybrid models, blending spiking neural networks (SNNs) and artificial neural networks (ANNs) using PyTorch and Lava frameworks. Our hybrid architecture integrates an SNN for temporal feature extraction and an ANN for classification. We delve into the challenges of deploying such hybrid structures on hardware. Specifically, we deploy individual components on Intel's Neuromorphic Processor Loihi (for SNN) and Jetson Nano (for ANN). We also propose an accumulator circuit to transfer data from the spiking to the non-spiking domain. Furthermore, we conduct comprehensive performance analyses of hybrid SNN-ANN models on a heterogeneous system of neuromorphic and edge AI hardware, evaluating accuracy, latency, power, and energy consumption. Our findings demonstrate that the hybrid spiking networks surpass the baseline ANN model across all metrics and outperform the baseline SNN model in accuracy and latency.
more »
« less
Photonic advantage of optical encoders
Abstract Light’s ability to perform massive linear operations in parallel has recently inspired numerous demonstrations of optics-assisted artificial neural networks (ANN). However, a clear system-level advantage of optics over purely digital ANN has not yet been established. While linear operations can indeed be optically performed very efficiently, the lack of nonlinearity and signal regeneration require high-power, low-latency signal transduction between optics and electronics. Additionally, a large power is needed for lasers and photodetectors, which are often neglected in the calculation of the total energy consumption. Here, instead of mapping traditional digital operations to optics, we co-designed a hybrid optical-digital ANN, that operates on incoherent light, and is thus amenable to operations under ambient light. Keeping the latency and power constant between a purely digital ANN and a hybrid optical-digital ANN, we identified a low-power/latency regime, where an optical encoder provides higher classification accuracy than a purely digital ANN. We estimate our optical encoder enables ∼10 kHz rate operation of a hybrid ANN with a power of only 23 mW. However, in that regime, the overall classification accuracy is lower than what is achievable with higher power and latency. Our results indicate that optics can be advantageous over digital ANN in applications, where the overall performance of the ANN can be relaxed to prioritize lower power and latency.
more »
« less
- PAR ID:
- 10477215
- Publisher / Repository:
- Nanophotonics
- Date Published:
- Journal Name:
- Nanophotonics
- Volume:
- 0
- Issue:
- 0
- ISSN:
- 2192-8606
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Sub-wavelength diffractive meta-optics have emerged as a versatile platform to manipulate light fields at will, due to their ultra-small form factor and flexible multifunctionalities. However, miniaturization and multimodality are typically compromised by a reduction in imaging performance; thus, meta-optics often yield lower resolution and stronger aberration compared to traditional refractive optics. Concurrently, computational approaches have become popular to improve the image quality of traditional cameras and exceed limitations posed by refractive lenses. This in turn often comes at the expense of higher power and latency, and such systems are typically limited by the availability of certain refractive optics. Limitations in both fields have thus sparked cross-disciplinary efforts to not only overcome these roadblocks but also to go beyond and provide synergistic meta-optical–digital solutions that surpass the potential of the individual components. For instance, an application-specific meta-optical frontend can preprocess the light field of a scene and focus it onto the sensor with a desired encoding, which can either ease the computational load on the digital backend or can intentionally alleviate certain meta-optical aberrations. In this review, we introduce the fundamentals, summarize the development of meta-optical computational imaging, focus on latest advancements that redefine the current state of the art, and give a perspective on research directions that leverage the full potential of sub-wavelength photonic platforms in imaging and sensing applications. The current advancement of meta-optics and recent investments by foundries and technology partners have the potential to provide synergistic future solutions for highly efficient, compact, and low-power imaging systems.more » « less
-
A feedforward optical neural network for image recognition is demonstrated through free-space optics in which the nonlinear activation function is provided by a resonant semiconductor saturable absorption mirror (RSAM). Although free-space optics with liquid crystal based spatial light modulator has the potential to allow parallel processing for improved performance, using a fiber-optic electrooptic modulator for parallel to serial conversion greatly simplifies the experimental setup. We show that this nonlinear activation function introduced by RSAM can provide a notable improvement of 8.1% to classification accuracy in comparison to a purely linear network when tested with the MNIST data set for image classification. The impact of noise is also investigated in system implementation.more » « less
-
Abstract Energy efficiency in computation is ultimately limited by noise, with quantum limits setting the fundamental noise floor. Analog physical neural networks hold promise for improved energy efficiency compared to digital electronic neural networks. However, they are typically operated in a relatively high-power regime so that the signal-to-noise ratio (SNR) is large (>10), and the noise can be treated as a perturbation. We study optical neural networks where all layers except the last are operated in the limit that each neuron can be activated by just a single photon, and as a result the noise on neuron activations is no longer merely perturbative. We show that by using a physics-based probabilistic model of the neuron activations in training, it is possible to perform accurate machine-learning inference in spite of the extremely high shot noise (SNR ~ 1). We experimentally demonstrated MNIST handwritten-digit classification with a test accuracy of 98% using an optical neural network with a hidden layer operating in the single-photon regime; the optical energy used to perform the classification corresponds to just 0.038 photons per multiply-accumulate (MAC) operation. Our physics-aware stochastic training approach might also prove useful with non-optical ultra-low-power hardware.more » « less
-
Rapid advancements in autonomous systems and the Internet of Things have necessitated the development of compact and low-power image sensors to bridge the gap between the digital and physical world. To that end, sub-wavelength diffractive optics, commonly known as meta-optics, have garnered significant interest from the optics and photonics community due to their ability to achieve multiple functionalities within a small form factor. Despite years of research, however, the performance of meta-optics has often remained inferior compared to that of traditional refractive optics. In parallel, computational imaging techniques have emerged as a promising path to miniaturize optical systems, albeit often at the expense of higher power and latency. The lack of desired performance from either meta-optical or computational solutions has motivated researchers to look into a jointly optimized meta-optical–digital solution. While the meta-optical front end can preprocess the scene to reduce the computational load on the digital back end, the computational back end can in turn relax requirements on the meta-optics. In this Perspective, we provide an overview of this up-and-coming field, termed here as “software-defined meta-optics.” We highlight recent contributions that have advanced the current state of the art and point out directions toward which future research efforts should be directed to leverage the full potential of subwavelength photonic platforms in imaging and sensing applications. Synergistic technology transfer and commercialization of meta-optic technologies will pave the way for highly efficient, compact, and low-power imaging systems of the future.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1515/nanoph-2023-0579
