skip to main content

Title: Sparse Coding Enables the Reconstruction of High-Fidelity Images and Video from Retinal Spike Trains
The optic nerve transmits visual information to the brain as trains of discrete events, a low-power, low-bandwidth communication channel also exploited by silicon retina cameras. Extracting highfidelity visual input from retinal event trains is thus a key challenge for both computational neuroscience and neuromorphic engineering. Here, we investigate whether sparse coding can enable the reconstruction of high-fidelity images and video from retinal event trains. Our approach is analogous to compressive sensing, in which only a random subset of pixels are transmitted and the missing information is estimated via inference. We employed a variant of the Locally Competitive Algorithm to infer sparse representations from retinal event trains, using a dictionary of convolutional features optimized via stochastic gradient descent and trained in an unsupervised manner using a local Hebbian learning rule with momentum. We used an anatomically realistic retinal model with stochastic graded release from cones and bipolar cells to encode thumbnail images as spike trains arising from ON and OFF retinal ganglion cells. The spikes from each model ganglion cell were summed over a 32 msec time window, yielding a noisy rate-coded image. Analogous to how the primary visual cortex is postulated to infer features from noisy spike trains arising from the optic nerve, we inferred a higher-fidelity sparse reconstruction from the noisy rate-coded image using a convolutional dictionary trained on the original CIFAR10 database. more » To investigate whether a similar approachworks on non-stochastic data, we demonstrate that the same procedure can be used to reconstruct high-frequency video from the asynchronous events arising from a silicon retina camera moving through a laboratory environment. « less
; ; ;
Award ID(s):
Publication Date:
Journal Name:
ICONS '18 Proceedings of the International Conference on Neuromorphic Systems Article No. 8
Page Range or eLocation-ID:
1 to 5
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations formore »accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.« less
  2. Bioengineering systems have transformed scientific knowledge of cellular behaviors in the nervous system (NS) and pioneered innovative, regenerative therapies to treat adult neural disorders. Microscale systems with characteristic lengths of single to hundreds of microns have examined the development and specialized behaviors of numerous neuromuscular and neurosensory components of the NS. The visual system is comprised of the eye sensory organ and its connecting pathways to the visual cortex. Significant vision loss arises from dysfunction in the retina, the photosensitive tissue at the eye posterior that achieves phototransduction of light to form images in the brain. Retinal regenerative medicine hasmore »embraced microfluidic technologies to manipulate stem-like cells for transplantation therapies, where de/differentiated cells are introduced within adult tissue to replace dysfunctional or damaged neurons. Microfluidic systems coupled with stem cell biology and biomaterials have produced exciting advances to restore vision. The current article reviews contemporary microfluidic technologies and microfluidics-enhanced bioassays, developed to interrogate cellular responses to adult retinal cues. The focus is on applications of microfluidics and microscale assays within mammalian sensory retina, or neuro retina, comprised of five types of retinal neurons (photoreceptors, horizontal, bipolar, amacrine, retinal ganglion) and one neuroglia (Müller), but excludes the non-sensory, retinal pigmented epithelium.« less
  3. Convolutional neural networks (CNNs), a class of deep learning models, have experienced recent success in modeling sensory cortices and retinal circuits through optimizing performance on machine learning tasks, otherwise known as task optimization. Previous research has shown task-optimized CNNs to be capable of providing explanations as to why the retina efficiently encodes natural stimuli and how certain retinal cell types are involved in efficient encoding. In our work, we sought to use task-optimized CNNs as a means of explaining computational mechanisms responsible for motion-selective retinal circuits. We designed a biologically constrained CNN and optimized its performance on a motion-classification task.more »We drew inspiration from psychophysics, deep learning, and systems neuroscience literature to develop a toolbox of methods to reverse engineer the computational mechanisms learned in our model. Through reverse engineering our model, we proposed a computational mechanism in which direction-selective ganglion cells and starburst amacrine cells, both experimentally observed retinal cell types, emerge in our model to discriminate among moving stimuli. This emergence suggests that direction-selective circuits in the retina are ecologically designed to robustly discriminate among moving stimuli. Our results and methods also provide a framework for how to build more interpretable deep learning models and how to understand them.« less
  4. Telecystoscopy can lower the barrier to access critical urologic diagnostics for patients around the world. A major challenge for robotic control of flexible cystoscopes and intuitive teleoperation is the pose estimation of the scope tip. We propose a novel real-time camera localization method using video recordings from a prior cystoscopy and 3D bladder reconstruction to estimate cystoscope pose within the bladder during follow-up telecystoscopy. We map prior video frames into a low-dimensional space as a dictionary so that a new image can be likewise mapped to efficiently retrieve its nearest neighbor among the dictionary images. The cystoscope pose is thenmore »estimated by the correspondence among the new image, its nearest dictionary image, and the prior model from 3D reconstruction. We demonstrate performance of our methods using bladder phantoms with varying fidelity and a servo-controlled cystoscope to simulate the use case of bladder surveillance through telecystoscopy. The servo-controlled cystoscope with 3 degrees of freedom (angulation, roll, and insertion axes) was developed for collecting cystoscope videos from bladder phantoms. Cystoscope videos were acquired in a 2.5D bladder phantom (bladder-shape cross-section plus height) with a panorama of a urothelium attached to the inner surface. Scans of the 2.5D phantom were performed in separate arc trajectories each of which is generated by actuation on the angulation with a fixed roll and insertion length. We further included variance in moving speed, imaging distance and existence of bladder tumors. Cystoscope videos were also acquired in a water-filled 3D silicone bladder phantom with hand-painted vasculature. Scans of the 3D phantom were performed in separate circle trajectories each of which is generated by actuation on the roll axis under a fixed angulation and insertion length. These videos were used to create 3D reconstructions, dictionary sets, and test data sets for evaluating the computational efficiency and accuracy of our proposed method in comparison with a method based on global Scale-Invariant Feature Transform (SIFT) features, named SIFT-only. Our method can retrieve the nearest dictionary image for 94–100% of test frames in under 55[Formula: see text]ms per image, whereas the SIFT-only method can only find the image match for 56–100% of test frames in 6000–40000[Formula: see text]ms per image depending on size of the dictionary set and richness of SIFT features in the images. Our method, with a speed of around 20 Hz for the retrieval stage, is a promising tool for real-time image-based scope localization in robotic cystoscopy when prior cystoscopy images are available.« less
  5. The choroid provides the majority of blood flow to the ocular tissues and structures that facilitate the processes of retinal metabolism responsible for vision. Specifically, the choriocapillaris provides a structural network of small blood vessels that supplies the retinal ganglion cells and deep ocular tissues. Similar to retinal nerve fiber layer thickness, choroidal thickness (CT) has been suggested to represent a quantifiable health biomarker for choroidal tissues. Glaucoma is a disease with vascular contributions in its onset and progression. Despite its importance in maintaining ocular structure and vascular functionality, clinical assessments of choroidal tissues have been historically challenged by themore »inaccessibility of CT biomarker targets. The development of optical coherence tomography angiography and enhanced depth imaging created a framework for assessing CT and investigating its relationship to glaucomatous optic neuropathy onset and progression. Pilot studies on CT in glaucoma are conflicting—with those both in support of, and against, its clinical utility. Complicating the data are highly customized analysis methods, small sample sizes, heterogeneous patient groups, and a lack of properly designed controlled studies with CT as a primary outcome. Herein, we review the available data on CT and critically discuss its potential relevance and limitations in glaucoma disease management.« less