skip to main content

Title: Curved neuromorphic image sensor array using a MoS2-organic heterostructure inspired by the human visual recognition system

Conventional imaging and recognition systems require an extensive amount of data storage, pre-processing, and chip-to-chip communications as well as aberration-proof light focusing with multiple lenses for recognizing an object from massive optical inputs. This is because separate chips (i.e., flat image sensor array, memory device, and CPU) in conjunction with complicated optics should capture, store, and process massive image information independently. In contrast, human vision employs a highly efficient imaging and recognition process. Here, inspired by the human visual recognition system, we present a novel imaging device for efficient image acquisition and data pre-processing by conferring the neuromorphic data processing function on a curved image sensor array. The curved neuromorphic image sensor array is based on a heterostructure of MoS2and poly(1,3,5-trimethyl-1,3,5-trivinyl cyclotrisiloxane). The curved neuromorphic image sensor array features photon-triggered synaptic plasticity owing to its quasi-linear time-dependent photocurrent generation and prolonged photocurrent decay, originated from charge trapping in the MoS2-organic vertical stack. The curved neuromorphic image sensor array integrated with a plano-convex lens derives a pre-processed image from a set of noisy optical inputs without redundant data storage, processing, and communications as well as without complex optics. The proposed imaging device can substantially improve efficiency of the image acquisition more » and recognition process, a step forward to the next generation machine vision.

« less
; ; ; ; ; ; ; ; ; ; ; ; ;
Award ID(s):
1720633 1904216 1708852 1935775
Publication Date:
Journal Name:
Nature Communications
Nature Publishing Group
Sponsoring Org:
National Science Foundation
More Like this
  1. An eight-element oil-filled hydrophone array is used to measure the acoustic field in littoral waters. This prototype array was deployed during an experiment between Jeffrey’s Ledge and the Stellwagen Bank region off the coast of Rockport, Massachusetts USA. During the experiment, several humpback whale vocalizations, distant ship tonals and high frequency conventional echosounder pings were recorded. Visual confirmation of humpback moving in bearing relative to the array verifies the directional sensing from array beamforming. During deployment, the array is towed at speeds varying from 4-7 kts in water depths of roughly 100 m with conditions at sea state 2 to 3. This array system consists of a portable winch with array, tow cable and 3 water-resistant boxes housing electronics. This system is deployed and operated by 2 crew members onboard a 13 m commercial fishing vessel during the experiment. Non-acoustic sensor (NAS) information is obtained to provide depth, temperature, and heading data using commercial off the shelf (COTS) components utilizing RS485/232 data communications. Acoustic data sampling was performed at 8 kHz, 30 kHz and 100 kHz with near real-time processing of data and enhanced Signal to Noise Ratio (SNR) from beamforming. The electrical system components are deployed with 3 stackedmore »electronics boxes housing power, data acquisition and data processing components in water resistant compartments. A laptop computer with 8 TB of external storage and an independent Global Positioning System (GPS) antenna is used to run Passive Ocean Acoustic Waveguide Remote Sensing (POAWRS) software providing beamformed spectrogram data and live NAS data with capability of capturing several days of data. The acquisition system consists of Surface Mount Device (SMD) pre-amplifiers with filter to an analog differential pair shipboard COTS acquisition system. Pre-amplifiers are constructed using SMD technology where components are pressure tolerant and potting is not necessary. Potting of connectors, electronics and hydrophones via 3D printed molding techniques will be discussed. Array internal components are manufactured with Thermoplastic Polyurethane (TPU) 3D printed material to dampen array vibrations with forward and aft vibration isolation modules (VIM). Polyurethane foam (PUF) used to scatter breathing waves and dampen contact from wires inside the array without attenuating high frequencies and allowing for significant noise reduction. A single Tygon array section with a length of 7.5 m and diameter of 38 mm contains 8 transducer elements with a spacing of 75 cm (1 kHz design frequency). Pre- amplifiers and NAS modules are affixed using Vectran and steel wire rope positioned by swaged stops along the strength member. The tow cable length is 100 m with a diameter of 22 mm that is potted to a hose adapter to break out 12 braided copper wire twisted pair conductors and terminates the tow cable Vectran braid. This array in its current state of development is a low-cost alternative to obtain quality acoustic data from a towed array system. Used here for observation of whale vocalizations, this type of array also has many applications in military sonar and seismic surveying. Maintenance on the array can be performed without the use of special facilities or equipment for dehosing and conveniently uses castor oil as an environmentally safe pressure compensating and coupling fluid. Array development including selection of transducers, NAS modules, acoustic acquisition system, array materials and method of construction with results from several deployments will be discussed. We also present beamformed spectrograms containing humpback whale downsweep moans and underwater blowing (bubbles) sounds associated with feeding on sand lance (Ammodytes dubius).« less
  2. Abstract

    As machine vision technology generates large amounts of data from sensors, it requires efficient computational systems for visual cognitive processing. Recently, in-sensor computing systems have emerged as a potential solution for reducing unnecessary data transfer and realizing fast and energy-efficient visual cognitive processing. However, they still lack the capability to process stored images directly within the sensor. Here, we demonstrate a heterogeneously integrated 1-photodiode and 1 memristor (1P-1R) crossbar for in-sensor visual cognitive processing, emulating a mammalian image encoding process to extract features from the input images. Unlike other neuromorphic vision processes, the trained weight values are applied as an input voltage to the image-saved crossbar array instead of storing the weight value in the memristors, realizing the in-sensor computing paradigm. We believe the heterogeneously integrated in-sensor computing platform provides an advanced architecture for real-time and data-intensive machine-vision applications via bio-stimulus domain reduction.

  3. Fast electrochemical imaging enables the dynamic study of electroactive molecule diffusion in neurotransmitter release from single cells and dopamine mapping in brain slices. In this paper, we discuss the design of an electrochemical imaging sensor using a monolithic CMOS sensor array and a multifunctional data acquisition system. Using post-CMOS fabrication, the CMOS sensor integrates 1024 on-chip electrodes on the surface and contains 1024 low-noise amplifiers to simultaneous process parallel electrochemical recordings. Each electrochemical electrode and amplifier are optimized to operate at 10.38 kHz sampling rate. To support the operation of the high-throughput CMOS device, a multifunctional data acquisition device is developed to provide the required speed and accuracy. The high analog data rate of 10.63 MHz from all 1024 amplifiers is redundantly sampled by the custom-designed data acquisition system which can process up to 73.6 MHz with up to ~400 Mbytes/s data rate to a computer using USB 3.0 interface. To contain the liquid above the electrochemical sensors and prevent electronic and wire damage, we packaged the monolithic sensor using a 3D-printed well. Using the presented device, 32 pixel × 32 pixel electrochemical imaging of dopamine diffusion is successfully demonstrated at over 10,000 frames per second, the fastest reported tomore »date.« less
  4. The traditional von Neumann architecture limits the increase in computing efficiency and results in massive power consumption in modern computers due to the separation of storage and processing units. The novel neuromorphic computation system, an in-memory computing architecture with low power consumption, is aimed to break the bottleneck and meet the needs of the next generation of artificial intelligence (AI) systems. Thus, it is urgent to find a memory technology to implement the neuromorphic computing nanosystem. Nowadays, the silicon-based flash memory dominates non-volatile memory market, however, it is facing challenging issues to achieve the requirements of future data storage device development due to the drawbacks, such as scaling issue, relatively slow operation speed, and high voltage for program/erase operations. The emerging resistive random-access memory (RRAM) has prompted extensive research as its simple two-terminal structure, including top electrode (TE) layer, bottom electrode (BE) layer, and an intermediate resistive switching (RS) layer. It can utilize a temporary and reversible dielectric breakdown to cause the RS phenomenon between the high resistance state (HRS) and the low resistance state (LRS). RRAM is expected to outperform conventional memory device with the advantages, notably its low-voltage operation, short programming time, great cyclic stability, and good scalability.more »Among the materials for RS layer, indium gallium zinc oxide (IGZO) has shown attractive prospects in abundance and high atomic diffusion property of oxygen atoms, transparency. Additionally, its electrical properties can be easily modulated by controlling the stoichiometric ratio of indium and gallium as well as oxygen potential in the sputter gas. Moreover, since the IGZO can be applied to both the thin-film transistor (TFT) channel and RS layer, it has a great potential for fully integrated transparent electronics application. In this work, we proposed amorphous transparent IGZO-based RRAMs and investigated switching behaviors of the memory cells prepared with different top electrodes. First, ITO was choosing to serve as both TE and BE to achieve high transmittance. A multi-target magnetron sputtering system was employed to deposit all three layers (TE, RS, BE layers) on glass substrate. I-V characteristics were evaluated by a semiconductor parameter analyzer, and the bipolar RS feature of our RRAM devices was demonstrated by typical butterfly curves. The optical transmission analysis was carried out via a UV-Vis spectrometer and the average transmittance was around 80% out of entire devices in the visible-light wavelength range, implying high transparency. We adjusted the oxygen partial pressure during the sputtering of IGZO to optimize the property because the oxygen vacancy concentration governs the RS performance. Electrode selection is crucial and can impact the performance of the whole device. Thus, Cu TE was chosen for our second type of device because the diffusion of Cu ions can be beneficial for the formation of the conductive filament (CF). A ~5 nm SiO 2 barrier layer was employed between TE and RS layers to confine the diffusion of Cu into the RS layer. At the same time, this SiO 2 inserting layer can provide an additional interfacial series resistance in the device to lower the off current, consequently, improve the on/off ratio and whole performance. Finally, an oxygen affinity metal Ti was selected as the TE for our third type of device because the concentration of the oxygen atoms can be shifted towards the Ti electrode, which provides an oxygengettering activity near the Ti metal. This process may in turn lead to the formation of a sub-stoichiometric region in the neighboring oxide that is believed to be the origin of better performance. In conclusion, the transparent amorphous IGZO-based RRAMs were established. To tune the property of RS layer, the sputtering conditions of RS were varied. To investigate the influence of TE selections on switching performance of RRAMs, we integrated a set of TE materials, and a barrier layer on IGZO-based RRAM and compared the switch characteristics. Our encouraging results clearly demonstrate that IGZO is a promising material in RRAM applications and breaking the bottleneck of current memory technologies.« less
  5. We present an on-chip, widefield fluorescence microscope, which consists of a diffuser placed a few millimeters away from a traditional image sensor. The diffuser replaces the optics of a microscope, resulting in a compact and easy-to-assemble system with a practical working distance of over 1.5 mm. Furthermore, the diffuser encodes volumetric information, enabling refocusability in post-processing and three-dimensional (3D) imaging of sparse samples from a single acquisition. Reconstruction of images from the raw data requires a precise model of the system, so we introduce a practical calibration scheme and a physics-based forward model to efficiently account for the spatially-varying point spread function (PSF). To improve performance in low-light, we propose a random microlens diffuser, which consists of many small lenslets randomly placed on the mask surface and yields PSFs that are robust to noise. We build an experimental prototype and demonstrate our system on both planar and 3D samples.