Sparse Coding Enables the Reconstruction of High-Fidelity Images and Video from Retinal Spike Trains
The optic nerve transmits visual information to the brain as trains
of discrete events, a low-power, low-bandwidth communication
channel also exploited by silicon retina cameras. Extracting highfidelity
visual input from retinal event trains is thus a key challenge
for both computational neuroscience and neuromorphic engineering.
Here, we investigate whether sparse coding can enable the
reconstruction of high-fidelity images and video from retinal event
trains. Our approach is analogous to compressive sensing, in which
only a random subset of pixels are transmitted and the missing
information is estimated via inference. We employed a variant of
the Locally Competitive Algorithm to infer sparse representations
from retinal event trains, using a dictionary of convolutional features
optimized via stochastic gradient descent and trained in an
unsupervised manner using a local Hebbian learning rule with
momentum.
We used an anatomically realistic retinal model with stochastic
graded release from cones and bipolar cells to encode thumbnail
images as spike trains arising from ON and OFF retinal ganglion
cells. The spikes from each model ganglion cell were summed over a
32 msec time window, yielding a noisy rate-coded image. Analogous
to how the primary visual cortex is postulated to infer features
from noisy spike trains arising from the optic nerve, we inferred
a higher-fidelity sparse reconstruction from the noisy rate-coded
image using a convolutional dictionary trained on the original
CIFAR10 database. To investigate whether a similar approachworks on non-stochastic
data, we demonstrate that the same procedure can be used to reconstruct
high-frequency video from the asynchronous events arising from a silicon retina camera moving through a laboratory environment.
more »
« less
- Award ID(s):
- 1734980
- NSF-PAR ID:
- 10075673
- Date Published:
- Journal Name:
- ICONS '18 Proceedings of the International Conference on Neuromorphic Systems Article No. 8
- Page Range / eLocation ID:
- 1 to 5
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
more » « less
Abstract Objective . Patients with photovoltaic subretinal implant PRIMA demonstrated letter acuity ∼0.1 logMAR worse than sampling limit for 100μ m pixels (1.3 logMAR) and performed slower than healthy subjects tested with equivalently pixelated images. To explore the underlying differences between natural and prosthetic vision, we compare the fidelity of retinal response to visual and subretinal electrical stimulation through single-cell modeling and ensemble decoding.Approach . Responses of retinal ganglion cells (RGCs) to optical or electrical white noise stimulation in healthy and degenerate rat retinas were recorded via multi-electrode array. Each RGC was fit with linear–nonlinear and convolutional neural network models. To characterize RGC noise, we compared statistics of spike-triggered averages (STAs) in RGCs responding to electrical or visual stimulation of healthy and degenerate retinas. At the population level, we constructed a linear decoder to determine the accuracy of the ensemble of RGCs onN -way discrimination tasks.Main results . Although computational models can match natural visual responses well (correlation ∼0.6), they fit significantly worse to spike timings elicited by electrical stimulation of the healthy retina (correlation ∼0.15). In the degenerate retina, response to electrical stimulation is equally bad. The signal-to-noise ratio of electrical STAs in degenerate retinas matched that of the natural responses when 78 ± 6.5% of the spikes were replaced with random timing. However, the noise in RGC responses contributed minimally to errors in ensemble decoding. The determining factor in accuracy of decoding was the number of responding cells. To compensate for fewer responding cells under electrical stimulation than in natural vision, more presentations of the same stimulus are required to deliver sufficient information for image decoding.Significance . Slower-than-natural pattern identification by patients with the PRIMA implant may be explained by the lower number of electrically activated cells than in natural vision, which is compensated by a larger number of the stimulus presentations. -
Primary open-angle glaucoma (POAG) is an optic neuropathy characterized by irreversible retinal ganglion cell damage and visual field loss. The global POAG prevalence is estimated to be 3.05%, and near term is expected to significantly rise, especially within aging Asian populations. Primary angle-closure glaucoma disproportionately affects Asians, with up to four times greater prevalence of normal-tension glaucoma reported compared with high-tension glaucoma. Estimates for overall POAG prevalence in Asian populations vary, with Chinese and Indian populations representing the majority of future cases. Structural characteristics associated with glaucoma progression including the optic nerve head, retina, and cornea are distinct in Asians, serving as intermediates between African and European descent populations. Patterns in IOP suggest some similarities between races, with a significant inverse relationship between age and IOP only in Asian populations. Genetic differences have been suggested to play a role in these differences, however, a clear genetic pattern is yet to be established. POAG pathogenesis differs between Asians and other ethnicities, and it may differ within the broad classification of the Asian race. Greater awareness and further research are needed to improve treatment plans and outcomes for the increasingly high prevalence of normal tension glaucoma within aging Asian populations.more » « less
-
Purpose To investigate relationships between blood pressure and the thickness of single retinal layers in the macula. Methods Participants of the population-based Beijing Eye Study, free of retinal or optic nerve disease, underwent medical and ophthalmological examinations including optical coherence tomographic examination of the macula. Applying a multiple-surface segmentation solution, we automatically segmented the retina into its various layers. Results The study included 2237 participants (mean age 61.8±8.4 years, range 50–93 years). Mean thicknesses of the retinal nerve fibre layer (RNFL), ganglion cell layer (GCL), inner plexiform layer, inner nuclear layer (INL), outer plexiform layer, outer nuclear layer/external limiting membrane, ellipsoid zone, photoreceptor outer segments (POS) and retinal pigment epithelium–Bruch membrane were 31.1±2.3 µm, 39.7±3.5 µm, 38.4±3.3 µm, 34.8±2.0 µm, 28.1±3.0 µm, 79.2±7.3 µm, 22.9±0.6 µm, 19.2±3.3 µm and 20.7±1.4 µm, respectively. In multivariable analysis, higher systolic blood pressure (SBP) and diastolic blood pressure (DBP) were associated with thinner GCL and thicker INL, after adjusting for age, sex and axial length (all p<0.0056). Higher SBP was additionally associated with thinner POS and higher DBP with thinner RNFL. For an elevation of SBP/DBP by 10 mm Hg, the RNFL, GCL, INL and POS changed by 2.0, 3.0, 1.5 and 2.0 µm, respectively. Conclusions Thickness of RNFL, GCL and POS was inversely and INL thickness was positively associated with higher blood pressure, while the thickness of the other retinal layers was not significantly correlated with blood pressure. The findings may be helpful for refinement of the morphometric detection of retinal diseases.more » « less
-
more » « less
Abstract The mammalian retina encodes the visual world in action potentials generated by 20–50 functionally and anatomically-distinct types of retinal ganglion cell (RGC). Individual RGC types receive synaptic input from distinct presynaptic circuits; therefore, their responsiveness to specific features in the visual scene arises from the information encoded in synaptic input and shaped by postsynaptic signal integration and spike generation. Unfortunately, there is a dearth of tools for characterizing the computations reflected in RGC spike output. Therefore, we developed a statistical model, the separable Nonlinear Input Model, to characterize the excitatory and suppressive components of RGC receptive fields. We recorded RGC responses to a correlated noise (“cloud”) stimulus in an
in vitro preparation of mouse retina and found that our model accurately predicted RGC responses at high spatiotemporal resolution. It identified multiple receptive fields reflecting the main excitatory and suppressive components of the response of each neuron. Significantly, our model accurately identified ON-OFF cells and distinguished their distinct ON and OFF receptive fields, and it demonstrated a diversity of suppressive receptive fields in the RGC population. In total, our method offers a rich description of RGC computation and sets a foundation for relating it to retinal circuitry. -
Abstract Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3229884.3229892
