More Like this

1. Universality and individuality in neural dynamics across large populations of recurrent networks

   Maheswaranathan, N.; Williams, A; Golub, M; Ganguli, S; Sussillo, D (December 2019, Advances in neural information processing systems)

   null (Ed.)

   Many recent studies have employed task-based modeling with recurrent neural networks (RNNs) to infer the computational function of different brain regions. These models are often assessed by quantitatively comparing the low-dimensional neural dynamics of the model and the brain, for example using canonical correlation analysis (CCA). However, the nature of the detailed neurobiological inferences one can draw from such efforts remains elusive. For example, to what extent does training neural networks to solve simple tasks, prevalent in neuroscientific studies, uniquely determine the low-dimensional dynamics independent of neural architectures? Or alternatively, are the learned dynamics highly sensitive to different neural architectures? Knowing the answer to these questions has strong implications on whether and how to use task-based RNN modeling to understand brain dynamics. To address these foundational questions, we study populations of thousands of networks of commonly used RNN architectures trained to solve neuroscientifically motivated tasks and characterize their low-dimensional dynamics via CCA and nonlinear dynamical systems analysis. We find the geometry of the dynamics can be highly sensitive to different network architectures, and further find striking dissociations between geometric similarity as measured by CCA and network function, yielding a cautionary tale. Moreover, we find that while the geometry of neural dynamics can vary greatly across architectures, the underlying computational scaffold: the topological structure of fixed points, transitions between them, limit cycles, and linearized dynamics, often appears {\it universal} across all architectures. Overall, this analysis of universality and individuality across large populations of RNNs provides a much needed foundation for interpreting quantitative measures of dynamical similarity between RNN and brain dynamics. 

   more » « less
2. Hardware based spatio-temporal neural processing backend for imaging sensors: Towards a smart camera

   https://doi.org/10.1117/12.2305137  

   Gu, Yunfei; Ganguly, Samiran; Stan, Mircea R.; Ghosh, Avik W.; Dhar, Nibir K.; Dutta, Achyut K. (May 2018, Image Sensing Technologies: Materials, Devices, Systems, and Applications V)

   In this work we show how we can build a technology platform for cognitive imaging sensors using recent advances in recurrent neural network architectures and training methods inspired from biology. We demonstrate learning and processing tasks specific to imaging sensors, including enhancement of sensitivity and signal-to-noise ratio (SNR) purely through neural filtering beyond the fundamental limits sensor materials, and inferencing and spatio-temporal pattern recognition capabilities of these networks with applications in object detection, motion tracking and prediction. We then show designs of unit hardware cells built using complementary metal-oxide semiconductor (CMOS) and emerging materials technologies for ultra-compact and energy-efficient embedded neural processors for smart cameras. 

   more » « less
3. Design Space for Graph Neural Networks

   You, Jiaxuan; Ying, Rex; Leskovec, Jure. (January 2020, Neural Information Processing Systems (NeurIPS))

   The rapid evolution of Graph Neural Networks (GNNs) has led to a growing number of new architectures as well as novel applications. However, current research focuses on proposing and evaluating specific architectural designs of GNNs, such as GCN, GIN, or GAT, as opposed to studying the more general design space of GNNs that consists of a Cartesian product of different design dimensions, such as the number of layers or the type of the aggregation function. Additionally, GNN designs are often specialized to a single task, yet few efforts have been made to understand how to quickly find the best GNN design for a novel task or a novel dataset. Here we define and systematically study the architectural design space for GNNs which consists of 315,000 different designs over 32 different predictive tasks. Our approach features three key innovations: (1) A general GNN design space; (2) a GNN task space with a similarity metric, so that for a given novel task/dataset, we can quickly identify/transfer the best performing architecture; (3) an efficient and effective design space evaluation method which allows insights to be distilled from a huge number of model-task combinations. Our key results include: (1) A comprehensive set of guidelines for designing well-performing GNNs; (2) while best GNN designs for different tasks vary significantly, the GNN task space allows for transferring the best designs across different tasks; (3) models discovered using our design space achieve state-of-the-art performance. Overall, our work offers a principled and scalable approach to transition from studying individual GNN designs for specific tasks, to systematically studying the GNN design space and the task space. Finally, we release GraphGym, a powerful platform for exploring different GNN designs and tasks. GraphGym features modularized GNN implementation, standardized GNN evaluation, and reproducible and scalable experiment management 

   more » « less
4. Alleviating catastrophic forgetting using context-dependent gating and synaptic stabilization

   https://doi.org/10.1073/pnas.1803839115  

   Masse, Nicolas Y.; Grant, Gregory D.; Freedman, David J. (October 2018, Proceedings of the National Academy of Sciences)

   Humans and most animals can learn new tasks without forgetting old ones. However, training artificial neural networks (ANNs) on new tasks typically causes them to forget previously learned tasks. This phenomenon is the result of “catastrophic forgetting,” in which training an ANN disrupts connection weights that were important for solving previous tasks, degrading task performance. Several recent studies have proposed methods to stabilize connection weights of ANNs that are deemed most important for solving a task, which helps alleviate catastrophic forgetting. Here, drawing inspiration from algorithms that are believed to be implemented in vivo, we propose a complementary method: adding a context-dependent gating signal, such that only sparse, mostly nonoverlapping patterns of units are active for any one task. This method is easy to implement, requires little computational overhead, and allows ANNs to maintain high performance across large numbers of sequentially presented tasks, particularly when combined with weight stabilization. We show that this method works for both feedforward and recurrent network architectures, trained using either supervised or reinforcement-based learning. This suggests that using multiple, complementary methods, akin to what is believed to occur in the brain, can be a highly effective strategy to support continual learning. 

   more » « less
5. High-speed CMOS-free purely spintronic asynchronous recurrent neural network

   https://doi.org/10.1063/5.0129006  

   Mathews, Pranav_O; Duffee, Christian_B; Thayil, Abel; Stovall, Ty_E; Bennett, Christopher_H; Garcia-Sanchez, Felipe; Marinella, Matthew_J; Incorvia, Jean_Anne_C; Hassan, Naimul; Hu, Xuan; et al (March 2023, APL Machine Learning)

   The exceptional capabilities of the human brain provide inspiration for artificially intelligent hardware that mimics both the function and the structure of neurobiology. In particular, the recent development of nanodevices with biomimetic characteristics promises to enable the development of neuromorphic architectures with exceptional computational efficiency. In this work, we propose biomimetic neurons comprised of domain wall-magnetic tunnel junctions that can be integrated into the first trainable CMOS-free recurrent neural network with biomimetic components. This paper demonstrates the computational effectiveness of this system for benchmark tasks and its superior computational efficiency relative to alternative approaches for recurrent neural networks. 

   more » « less