An official website of the United States government
Abstract As machine vision technology generates large amounts of data from sensors, it requires efficient computational systems for visual cognitive processing. Recently, in-sensor computing systems have emerged as a potential solution for reducing unnecessary data transfer and realizing fast and energy-efficient visual cognitive processing. However, they still lack the capability to process stored images directly within the sensor. Here, we demonstrate a heterogeneously integrated 1-photodiode and 1 memristor (1P-1R) crossbar for in-sensor visual cognitive processing, emulating a mammalian image encoding process to extract features from the input images. Unlike other neuromorphic vision processes, the trained weight values are applied as an input voltage to the image-saved crossbar array instead of storing the weight value in the memristors, realizing the in-sensor computing paradigm. We believe the heterogeneously integrated in-sensor computing platform provides an advanced architecture for real-time and data-intensive machine-vision applications via bio-stimulus domain reduction.
more »
« less
Hetero‐Integrated InGaAs Photodiode and Oxide Memristor‐Based Artificial Optical Nerve for In‐Sensor NIR Image Processing
Abstract In‐sensor computing is an emerging architectural paradigm that fuses data acquisition and processing within a sensory domain. The integration of multiple functions into a single domain reduces the system footprint while it minimizes the energy and time for data transfer between sensory and computing units. However, it is challenging for a simple and compact image sensor array to achieve both sensing and computing in each pixel. Here, this work demonstrates a focal plane array with a heterogeneously integrated one‐photodiode one‐resistor (1P‐1R)‐based artificial optical neuron that emulates the sensing, computing, and memorization of a biological retina system. This work employs an InGaAs photodiode featuring a high responsivity and a broad spectrum that covers near‐infrared (NIR) signals and employs an HfO2memristor as the artificial synapse to achieve the computing/memorization in an analog domain. Using the fabricated focal plane array integrated with an artificial neural network, this work performs in‐sensor image identification of finger veins driven by NIR light illumination (≈84 % accuracy). The proposed in‐sensor image computing architecture that broadly covers the NIR spectrum offers widespread application of focal plane array for computer vision, neuromorphic computing, biomedical engineering, etc.
more »
« less
- Award ID(s):
- 1942868
- PAR ID:
- 10395197
- Publisher / Repository:
- Wiley Blackwell (John Wiley & Sons)
- Date Published:
- Journal Name:
- Advanced Optical Materials
- Volume:
- 11
- Issue:
- 3
- ISSN:
- 2195-1071
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Schmidt, Dirk; Schreiber, Laura; Vernet, Elise (Ed.)The MMTO Adaptive optics exoPlanet characterization System (MAPS) is an ongoing upgrade to the 6.5-meter MMT Observatory on Mount Hopkins in Arizona. MAPS includes an upgraded adaptive secondary mirror (ASM), upgrades to the ARIES spectrograph, and a new AO system containing both an optical and near-infrared (NIR; 0.9-1.8 μm) pyramid wavefront sensor (PyWFS). The NIR PyWFS will utilize an IR-optimized double pyramid coupled with a SAPHIRA detector: a low-read noise electron Avalanche Photodiode (eAPD) array. This NIR PyWFS will improve MAPS's sky coverage by an order of magnitude by allowing redder guide stars (e.g. K & M-dwarfs or highly obscured stars in the Galactic plane) to be used. To date, the custom designed cryogenic SAPHIRA camera has been fully characterized and can reach sub-electron read noise at high avalanche gain. In order to test the performance of the camera in a closed-loop environment prior to delivery to the observatory, an AO testbed was designed and constructed. In addition to testing the SAPHIRA's performance, the testbed will be used to test and further develop the proposed on-sky calibration procedure for MMTO's ASM. We will report on the anticipated performance improvements from our NIR PyWFS, the SAPHIRA's closed-loop performance on our testbed, and the status of our ASM calibration procedure.more » « less
-
Abstract Conventional imaging and recognition systems require an extensive amount of data storage, pre-processing, and chip-to-chip communications as well as aberration-proof light focusing with multiple lenses for recognizing an object from massive optical inputs. This is because separate chips (i.e., flat image sensor array, memory device, and CPU) in conjunction with complicated optics should capture, store, and process massive image information independently. In contrast, human vision employs a highly efficient imaging and recognition process. Here, inspired by the human visual recognition system, we present a novel imaging device for efficient image acquisition and data pre-processing by conferring the neuromorphic data processing function on a curved image sensor array. The curved neuromorphic image sensor array is based on a heterostructure of MoS2and poly(1,3,5-trimethyl-1,3,5-trivinyl cyclotrisiloxane). The curved neuromorphic image sensor array features photon-triggered synaptic plasticity owing to its quasi-linear time-dependent photocurrent generation and prolonged photocurrent decay, originated from charge trapping in the MoS2-organic vertical stack. The curved neuromorphic image sensor array integrated with a plano-convex lens derives a pre-processed image from a set of noisy optical inputs without redundant data storage, processing, and communications as well as without complex optics. The proposed imaging device can substantially improve efficiency of the image acquisition and recognition process, a step forward to the next generation machine vision.more » « less
-
Abstract Image sensors with internal computing capability enable in-sensor computing that can significantly reduce the communication latency and power consumption for machine vision in distributed systems and robotics. Two-dimensional semiconductors have many advantages in realizing such intelligent vision sensors because of their tunable electrical and optical properties and amenability for heterogeneous integration. Here, we report a multifunctional infrared image sensor based on an array of black phosphorous programmable phototransistors (bP-PPT). By controlling the stored charges in the gate dielectric layers electrically and optically, the bP-PPT’s electrical conductance and photoresponsivity can be locally or remotely programmed with 5-bit precision to implement an in-sensor convolutional neural network (CNN). The sensor array can receive optical images transmitted over a broad spectral range in the infrared and perform inference computation to process and recognize the images with 92% accuracy. The demonstrated bP image sensor array can be scaled up to build a more complex vision-sensory neural network, which will find many promising applications for distributed and remote multispectral sensing.more » « less
-
Abstract There is a long history of using angle sensors to measure wavefront. The best example is the Shack-Hartmann sensor. Compared to other methods of wavefront sensing, angle-based approach is more broadly used in industrial applications and scientific research. Its wide adoption is attributed to its fully integrated setup, robustness, and fast speed. However, there is a long-standing issue in its low spatial resolution, which is limited by the size of the angle sensor. Here we report a angle-based wavefront sensor to overcome this challenge. It uses ultra-compact angle sensor built from flat optics. It is directly integrated on focal plane array. This wavefront sensor inherits all the benefits of the angle-based method. Moreover, it improves the spatial sampling density by over two orders of magnitude. The drastically improved resolution allows angle-based sensors to be used for quantitative phase imaging, enabling capabilities such as video-frame recording of high-resolution surface topography.more » « less
