skip to main content

Title: Deep inference of latent dynamics with spatio-temporal super resolution using selective backpropagation through time
Modern neural interfaces allow access to the activity of up to a million neurons within brain circuits. However, bandwidth limits often create a trade-off between greater spatial sampling (more channels or pixels) and the temporal frequency of sampling. Here we demonstrate that it is possible to obtain spatio-temporal super-resolution in neuronal time series by exploiting relationships among neurons, embedded in latent low-dimensional population dynamics. Our novel neural network training strategy, selective backpropagation through time (SBTT), enables learning of deep generative models of latent dynamics from data in which the set of observed variables changes at each time step. The resulting models are able to infer activity for missing samples by combining observations with learned latent dynamics. We test SBTT applied to sequential autoencoders and demonstrate more efficient and higher-fidelity characterization of neural population dynamics in electrophysiological and calcium imaging data. In electrophysiology, SBTT enables accurate inference of neuronal population dynamics with lower interface bandwidths, providing an avenue to significant power savings for implanted neuroelectronic interfaces. In applications to two-photon calcium imaging, SBTT accurately uncovers high-frequency temporal structure underlying neural population activity, substantially outperforming the current state-of-the-art. Finally, we demonstrate that performance could be further improved by using limited, high-bandwidth sampling more » to pretrain dynamics models, and then using SBTT to adapt these models for sparsely-sampled data. « less
; ; ; ; ; ; ;
Award ID(s):
Publication Date:
Journal Name:
Advances in Neural Information Processing Systems (NeurIPS)
Sponsoring Org:
National Science Foundation
More Like this
  1. Abstract Objective . Neural prosthetics often use intracortical microstimulation (ICMS) for sensory restoration. To restore natural and functional feedback, we must first understand how stimulation parameters influence the recruitment of neural populations. ICMS waveform asymmetry modulates the spatial activation of neurons around an electrode at 10 Hz; however, it is unclear how asymmetry may differentially modulate population activity at frequencies typically employed in the clinic (e.g. 100 Hz). We hypothesized that stimulation waveform asymmetry would differentially modulate preferential activation of certain neural populations, and the differential population activity would be frequency-dependent. Approach . We quantified how asymmetric stimulation waveforms delivered at 10 or 100 Hz for 30 s modulated spatiotemporal activity of cortical layer II/III pyramidal neurons using in vivo two-photon and mesoscale calcium imaging in anesthetized mice. Asymmetry is defined in terms of the ratio of the duration of the leading phase to the duration of the return phase of charge-balanced cathodal- and anodal-first waveforms (i.e. longer leading phase relative to return has larger asymmetry). Main results . Neurons within 40–60 µ m of the electrode display stable stimulation-induced activity indicative of direct activation, which was independent of waveform asymmetry. The stability of 72% of activated neurons and themore »preferential activation of 20%–90% of neurons depended on waveform asymmetry. Additionally, this asymmetry-dependent activation of different neural populations was associated with differential progression of population activity. Specifically, neural activity tended to increase over time during 10 Hz stimulation for some waveforms, whereas activity remained at the same level throughout stimulation for other waveforms. During 100 Hz stimulation, neural activity decreased over time for all waveforms, but decreased more for the waveforms that resulted in increasing neural activity during 10 Hz stimulation. Significance. These data demonstrate that at frequencies commonly used for sensory restoration, stimulation waveform alters the pattern of activation of different but overlapping populations of excitatory neurons. The impact of these waveform specific responses on the activation of different subtypes of neurons as well as sensory perception merits further investigation.« less
  2. Abstract

    Dynamic community detection provides a coherent description of network clusters over time, allowing one to track the growth and death of communities as the network evolves. However, modularity maximization, a popular method for performing multilayer community detection, requires the specification of an appropriate null network as well as resolution and interlayer coupling parameters. Importantly, the ability of the algorithm to accurately detect community evolution is dependent on the choice of these parameters. In functional temporal networks, where evolving communities reflect changing functional relationships between network nodes, it is especially important that the detected communities reflect any state changes of the system. Here, we present analytical work suggesting that a uniform null network provides improved sensitivity to the detection of small evolving communities in temporal networks with positive edge weights bounded above by 1, such as certain types of correlation networks. We then propose a method for increasing the sensitivity of modularity maximization to state changes in nodal dynamics by modelling self-identity links between layers based on the self-similarity of the network nodes between layers. This method is more appropriate for functional temporal networks from both a modelling and mathematical perspective, as it incorporates the dynamic nature of network nodes.more »We motivate our method based on applications in neuroscience where network nodes represent neurons and functional edges represent similarity of firing patterns in time. We show that in simulated data sets of neuronal spike trains, updating interlayer links based on the firing properties of the neurons provides superior community detection of evolving network structure when groups of neurons change their firing properties over time. Finally, we apply our method to experimental calcium imaging data that monitors the spiking activity of hundreds of neurons to track the evolution of neuronal communities during a state change from the awake to anaesthetized state.

    « less
  3. Continuing advances in neural interfaces have enabled simultaneous monitoring of spiking activity from hundreds to thousands of neurons. To interpret these large-scale data, several methods have been proposed to infer latent dynamic structure from high-dimensional datasets. One recent line of work uses recurrent neural networks in a sequential autoencoder (SAE) framework to uncover dynamics. SAEs are an appealing option for modeling nonlinear dynamical systems, and enable a precise link between neural activity and behavior on a single-trial basis. However, the very large parameter count and complexity of SAEs relative to other models has caused concern that SAEs may only perform well on very large training sets. We hypothesized that with a method to systematically optimize hyperparameters (HPs), SAEs might perform well even in cases of limited training data. Such a breakthrough would greatly extend their applicability. However, we find that SAEs applied to spiking neural data are prone to a particular form of overfitting that cannot be detected using standard validation metrics, which prevents standard HP searches. We develop and test two potential solutions: an alternate validation method (“sample validation”) and a novel regularization method (“coordinated dropout”). These innovations prevent overfitting quite effectively, and allow us to test whether SAEsmore »can achieve good performance on limited data through large-scale HP optimization. When applied to data from motor cortex recorded while monkeys made reaches in various directions, large-scale HP optimization allowed SAEs to better maintain performance for small dataset sizes. Our results should greatly extend the applicability of SAEs in extracting latent dynamics from sparse, multidimensional data, such as neural population spiking activity.« less
  4. Abstract

    Multiphoton microscopy has emerged as the primary imaging tool for studying the structural and functional dynamics of neural circuits in brain tissue, which is highly scattering to light. Recently, three-photon microscopy has enabled high-resolution fluorescence imaging of neurons in deeper brain areas that lie beyond the reach of conventional two-photon microscopy, which is typically limited to ~ 450 µm. Three-photon imaging of neuronal calcium signals, through the genetically-encoded calcium indicator GCaMP6, has been used to successfully record neuronal activity in deeper neocortical layers and parts of the hippocampus in rodents. Bulk-loading cells in deeper cortical layers with synthetic calcium indicators could provide an alternative strategy for labelling that obviates dependence on viral tropism and promoter penetration, particularly in non-rodent species. Here we report a strategy for visualized injection of a calcium dye, Oregon Green BAPTA-1 AM (OGB-1 AM), at 500–600 µm below the surface of the mouse visual cortex in vivo. We demonstrate successful OGB-1 AM loading of cells in cortical layers 5–6 and subsequent three-photon imaging of orientation- and direction- selective visual responses from these cells.

  5. Gutkin, Boris S. (Ed.)
    Measuring the activity of neuronal populations with calcium imaging can capture emergent functional properties of neuronal circuits with single cell resolution. However, the motion of freely behaving animals, together with the intermittent detectability of calcium sensors, can hinder automatic monitoring of neuronal activity and their subsequent functional characterization. We report the development and open-source implementation of a multi-step cellular tracking algorithm (Elastic Motion Correction and Concatenation or EMC 2 ) that compensates for the intermittent disappearance of moving neurons by integrating local deformation information from detectable neurons. We demonstrate the accuracy and versatility of our algorithm using calcium imaging data from two-photon volumetric microscopy in visual cortex of awake mice, and from confocal microscopy in behaving Hydra , which experiences major body deformation during its contractions. We quantify the performance of our algorithm using ground truth manual tracking of neurons, along with synthetic time-lapse sequences, covering a wide range of particle motions and detectability parameters. As a demonstration of the utility of the algorithm, we monitor for several days calcium activity of the same neurons in layer 2/3 of mouse visual cortex in vivo , finding significant turnover within the active neurons across days, with only few neurons that remainedmore »active across days. Also, combining automatic tracking of single neuron activity with statistical clustering, we characterize and map neuronal ensembles in behaving Hydra , finding three major non-overlapping ensembles of neurons (CB, RP1 and RP2) whose activity correlates with contractions and elongations. Our results show that the EMC 2 algorithm can be used as a robust and versatile platform for neuronal tracking in behaving animals.« less