More Like this

1. An Integrated Transportation Distance between Kernels and Approximate Dynamic Risk Evaluation in Markov Systems

   https://doi.org/10.1137/22M1530665  

   Lin, Zhengqi; Ruszczyński, Andrzej (December 2023, SIAM Journal on Control and Optimization)

   N/A (Ed.)

   We introduce a distance between kernels based on the Wasserstein distances between their values, study its properties, and prove that it is a metric on an appropriately defined space of kernels. We also relate it to various modes of convergence in the space of kernels. Then we consider the problem of approximating solutions to forward--backward systems, where the forward part is a Markov system described by a sequence of kernels, and the backward part calculates the values of a risk measure by operators that may be nonlinear with respect to the system's kernels. We propose recursively approximating the forward system with the use of the integrated transportation distance between kernels and we estimate the error of the risk evaluation by the errors of individual kernel approximations. We illustrate the results on stopping problems and several well-known risk measures. Then we develop a particle-based numerical procedure, in which the approximate kernels have finite support sets. Finally, we illustrate the efficacy of the approach on the financial problem of pricing an American basket option. 

   more » « less
2. Approximating Back-propagation for a Biologically Plausible Local Learning Rule in Spiking Neural Networks

   https://doi.org/10.1145/3354265.3354275  

   Shrestha, Amar; Fang, Haowen; Wu, Qing; Qiu, Qinru (July 2019, International Conference on Neuromorphic Systems)

   Asynchronous event-driven computation and communication using spikes facilitate the realization of spiking neural networks (SNN) to be massively parallel, extremely energy efficient and highly robust on specialized neuromorphic hardware. However, the lack of a unified robust learning algorithm limits the SNN to shallow networks with low accuracies. Artificial neural networks (ANN), however, have the backpropagation algorithm which can utilize gradient descent to train networks which are locally robust universal function approximators. But backpropagation algorithm is neither biologically plausible nor neuromorphic implementation friendly because it requires: 1) separate backward and forward passes, 2) differentiable neurons, 3) high-precision propagated errors, 4) coherent copy of weight matrices at feedforward weights and the backward pass, and 5) non-local weight update. Thus, we propose an approximation of the backpropagation algorithm completely with spiking neurons and extend it to a local weight update rule which resembles a biologically plausible learning rule spike-timing-dependent plasticity (STDP). This will enable error propagation through spiking neurons for a more biologically plausible and neuromorphic implementation friendly backpropagation algorithm for SNNs. We test the proposed algorithm on various traditional and non-traditional benchmarks with competitive results. 

   more » « less
3. COMET: a novel memory-efficient deep learning training framework by using error-bounded lossy compression

   https://doi.org/10.14778/3503585.3503597  

   Jin, Sian; Zhang, Chengming; Jiang, Xintong; Feng, Yunhe; Guan, Hui; Li, Guanpeng; Song, Shuaiwen Leon; Tao, Dingwen (December 2021, Proceedings of the VLDB Endowment)

   Deep neural networks (DNNs) are becoming increasingly deeper, wider, and non-linear due to the growing demands on prediction accuracy and analysis quality. Training wide and deep neural networks require large amounts of storage resources such as memory because the intermediate activation data must be saved in the memory during forward propagation and then restored for backward propagation. However, state-of-the-art accelerators such as GPUs are only equipped with very limited memory capacities due to hardware design constraints, which significantly limits the maximum batch size and hence performance speedup when training large-scale DNNs. Traditional memory saving techniques either suffer from performance overhead or are constrained by limited interconnect bandwidth or specific interconnect technology. In this paper, we propose a novel memory-efficient CNN training framework (called COMET) that leverages error-bounded lossy compression to significantly reduce the memory requirement for training in order to allow training larger models or to accelerate training. Our framework purposely adopts error-bounded lossy compression with a strict error-controlling mechanism. Specifically, we perform a theoretical analysis on the compression error propagation from the altered activation data to the gradients, and empirically investigate the impact of altered gradients over the training process. Based on these analyses, we optimize the error-bounded lossy compression and propose an adaptive error-bound control scheme for activation data compression. Experiments demonstrate that our proposed framework can significantly reduce the training memory consumption by up to 13.5X over the baseline training and 1.8X over another state-of-the-art compression-based framework, respectively, with little or no accuracy loss. 

   more » « less
4. MERIT: A Sustainable DNN Accelerator Design with Photonic Phase-Change Memory

   https://doi.org/10.1109/TSUSC.2024.3521847  

   Li, Yuan; Louri, Ahmed; Karanth, Avinash (January 2025, IEEE Transactions on Sustainable Computing)

   The growing computational demands of deep learning have driven interest in analog neural networks using resistive memory and silicon photonics. However, these technologies face inherent limitations in computing parallelism when used independently. Photonic phase-change memory (PCM), which integrates photonics with PCM, overcomes these constraints by enabling simultaneous processing of multiple inputs encoded on different wavelengths, significantly enhancing parallel computation for deep neural network (DNN) inference and training. This paper presents MERIT, a sustainable DNN accelerator that capitalizes on the non-volatility of resistive memory and the high operating speed of photonic devices. MERIT enables seamless inference and training by loading weight kernels into photonic PCM arrays and selectively supplying light encoded with input features for the forward pass and loss gradients for the backward pass. We compare MERIT with state-of-the-art digital and analog DNN accelerators including TPU, DEAP, and PTC. Simulation results demonstrate that MERIT reduces execution time by 68% and energy consumption by 64% for inference, and reduces execution time by 79% and energy consumption by 84% for training. 

   more » « less
5. Two Routes to Scalable Credit Assignment without Weight Symmetry, International Conference on Machine

   Kunin, D.; Nayebi, A; Javier, S.; Ganguli, S; Bloom, J.; Yamins, D. (April 2020, Proceedings of Machine Learning Research)

   null (Ed.)

   The neural plausibility of backpropagation has long been disputed, primarily for its use of non-local weight transport — the biologically dubious requirement that one neuron instantaneously measure the synaptic weights of another. Until recently, attempts to create local learning rules that avoid weight transport have typically failed in the large-scale learning scenarios where backpropagation shines, e.g. ImageNet categorization with deep convolutional networks. Here, we investigate a recently proposed local learning rule that yields competitive performance with backpropagation and find that it is highly sensitive to metaparameter choices, requiring laborious tuning that does not transfer across network architecture. Our analysis indicates the underlying mathematical reason for this instability, allowing us to identify a more robust local learning rule that better transfers without metaparameter tuning. Nonetheless, we find a performance and stability gap between this local rule and backpropagation that widens with increasing model depth. We then investigate several non-local learning rules that relax the need for instantaneous weight transport into a more biologically-plausible "weight estimation" process, showing that these rules match state-of-the-art performance on deep networks and operate effectively in the presence of noisy updates. Taken together, our results suggest two routes towards the discovery of neural implementations for credit assignment without weight symmetry: further improvement of local rules so that they perform consistently across architectures and the identification of biological implementations for non-local learning mechanisms. 

   more » « less