More Like this

1. Deep Reservoir Networks with Learned Hidden Reservoir Weights using Direct Feedback Alignment

   Evanusa, Matthew ; Fermüller, Cornelia ; Aloimonos, Yiannis ( October 2020 , ArXivorg)

   null (Ed.)

   Deep Reservoir Computing has emerged as a new paradigm for deep learning, which is based around the reservoir computing principle of maintaining random pools of neurons combined with hierarchical deep learning. The reservoir paradigm reflects and respects the high degree of recurrence in biological brains, and the role that neuronal dynamics play in learning. However, one issue hampering deep reservoir network development is that one cannot backpropagate through the reservoir layers. Recent deep reservoir architectures do not learn hidden or hierarchical representations in the same manner as deep artificial neural networks, but rather concatenate all hidden reservoirs together to perform traditional regression. Here we present a novel Deep Reservoir Network for time series prediction and classification that learns through the non-differentiable hidden reservoir layers using a biologically-inspired backpropagation alternative called Direct Feedback Alignment, which resembles global dopamine signal broadcasting in the brain. We demonstrate its efficacy on two real world multidimensional time series datasets. 

   more » « less
2. A machine-learning approach for long-term prediction of experimental cardiac action potential time series using an autoencoder and echo state networks

   https://doi.org/10.1063/5.0087812  

   Shahi, Shahrokh ; Fenton, Flavio H. ; Cherry, Elizabeth M. ( June 2022 , Chaos: An Interdisciplinary Journal of Nonlinear Science)

   Computational modeling and experimental/clinical prediction of the complex signals during cardiac arrhythmias have the potential to lead to new approaches for prevention and treatment. Machine-learning (ML) and deep-learning approaches can be used for time-series forecasting and have recently been applied to cardiac electrophysiology. While the high spatiotemporal nonlinearity of cardiac electrical dynamics has hindered application of these approaches, the fact that cardiac voltage time series are not random suggests that reliable and efficient ML methods have the potential to predict future action potentials. This work introduces and evaluates an integrated architecture in which a long short-term memory autoencoder (AE) is integrated into the echo state network (ESN) framework. In this approach, the AE learns a compressed representation of the input nonlinear time series. Then, the trained encoder serves as a feature-extraction component, feeding the learned features into the recurrent ESN reservoir. The proposed AE-ESN approach is evaluated using synthetic and experimental voltage time series from cardiac cells, which exhibit nonlinear and chaotic behavior. Compared to the baseline and physics-informed ESN approaches, the AE-ESN yields mean absolute errors in predicted voltage 6–14 times smaller when forecasting approximately 20 future action potentials for the datasets considered. The AE-ESN also demonstrates less sensitivity to algorithmic parameter settings. Furthermore, the representation provided by the feature-extraction component removes the requirement in previous work for explicitly introducing external stimulus currents, which may not be easily extracted from real-world datasets, as additional time series, thereby making the AE-ESN easier to apply to clinical data. 

   more » « less
3. Detection Through Deep Neural Networks: A Reservoir Computing Approach for MIMO-OFDM Symbol Detection

   Kangjun Bai, Lingjia Liu ( November 2020 , IEEE International Conference on Computer-Aided Design)

   null (Ed.)

   The Reservoir Computing, a neural computing framework suited for temporal information processing, utilizes a dynamic reservoir layer for high-dimensional encoding, enhancing the separability of the network. In this paper, we exploit a Deep Learning (DL)-based detection strategy for Multiple-input, Multiple-output Orthogonal Frequency-Division Multiplexing (MIMO-OFDM) symbol detection. To be specific, we introduce a Deep Echo State Network (DESN), a unique hierarchical processing structure with multiple time intervals, to enhance the memory capacity and accelerate the detection efficiency. The resulting hardware prototype with the hybrid memristor-CMOS co-design provides in-memory computing and parallel processing capabilities, significantly reducing the hardware and power overhead. With the standard 180nm CMOS process and memristive synapses, the introduced DESN consumes merely 105mW of power consumption, exhibiting 16.7% power reduction compared to shallow ESN designs even with more dynamic layers and associated neurons. Furthermore, numerical evaluations demonstrate the advantages of the DESN over state-of-the-art detection techniques in the literate for MIMO-OFDM systems even with a very limited training set, yielding a 47.8% improvement against conventional symbol detection techniques. 

   more » « less
4. Reservoir computing for modeling and predicting stream chemistry

   https://doi.org/10.2166/hydro.2025.213  

   Allsup, Paden ; Abbott, Ben_W ; Brown, Brian ; Giraud-Carrier, Christophe ( March 2025 , Journal of Hydroinformatics)

   This paper explores the use of Echo State Networks (ESNs), a subset of reservoir computing, in modeling and predicting streamflow variability with a focus on biogeochemical patterns. Multiple ESNs were tested alongside a comparable long short-term memory model (LSTM), another deep learning model commonly used in time-series modeling, in the hope of finding a more robust streamflow chemistry predictor. Testing revealed that for our specific modeling of water temperature and dissolved oxygen (DO) levels, ESNs outperform LSTMs in both model fit and time necessary for training and testing. Our conclusions are that for hydrological tasks where data forms a chaotic time series, ESNs provide a useful and efficient alternative to LSTMs, being quicker to train, providing better results, and being easier to apply to the given task.

    

   more » « less
5. Hybrid macro/micro level backpropagation for training deep spiking neural networks

   Jin, Yingyezhe ; Zhang, Wenrui ; Li, Peng ( December 2018 , Advances in neural information processing systems)

   Spiking neural networks (SNNs) are positioned to enable spatio-temporal information processing and ultra-low power event-driven neuromorphic hardware. However, SNNs are yet to reach the same performances of conventional deep artificial neural networks (ANNs), a long-standing challenge due to complex dynamics and non-differentiable spike events encountered in training. The existing SNN error backpropagation (BP) methods are limited in terms of scalability, lack of proper handling of spiking discontinuities, and/or mismatch between the rate coded loss function and computed gradient. We present a hybrid macro/micro level backpropagation (HM2-BP) algorithm for training multi-layer SNNs. The temporal effects are precisely captured by the proposed spike-train level post-synaptic potential (S-PSP) at the microscopic level. The rate-coded errors are defined at the macroscopic level, computed and back-propagated across both macroscopic and microscopic levels. Different from existing BP methods, HM2-BP directly computes the gradient of the rate-coded loss function w.r.t tunable parameters. We evaluate the proposed HM2-BP algorithm by training deep fully connected and convolutional SNNs based on the static MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an accuracy level of 99:49% and 98:88% for MNIST and N-MNIST, respectively, outperforming the best reported performances obtained from the existing SNN BP algorithms. Furthermore, the HM2-BP produces the highest accuracies based on SNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the 16-speaker spoken English letters of TI46 Corpus [16], a challenging spatio-temporal speech recognition benchmark for which no prior success based on SNNs was reported. It also achieves competitive performances surpassing those of conventional deep learning models when dealing with asynchronous spiking streams. 

   more » « less