More Like this

1. The Small-World Effect for Interferometer Networks

   https://doi.org/10.1088/2632-072X/ad4c45  

   Krawciw, Benjamin; Carr, Lincoln D.; Diniz Behn, Cecilia (May 2024, Journal of Physics: Complexity)

   Abstract Complex network theory has focused on properties of networks with real-valued edge weights. However, in signal transfer networks, such as those representing the transfer of light across an interferometer, complex-valued edge weights are needed to represent the manipulation of the signal in both magnitude and phase. These complex-valued edge weights introduce interference into the signal transfer, but it is unknown how such interference affects network properties such as small-worldness. To address this gap, we have introduced a small-world interferometer network model with complex-valued edge weights and generalized existing network measures to define the interferometric clustering coefficient, the apparent path length, and the interferometric small-world coefficient. Using high-performance computing resources, we generated a large set of small-world interferometers over a wide range of parameters in system size, nearest-neighbor count, and edge-weight phase and computed their interferometric network measures. We found that the interferometric small-world coefficient depends significantly on the amount of phase on complex-valued edge weights: for small edge- weight phases, constructive interference led to a higher interferometric small-world coefficient; while larger edge-weight phases induced destructive interference which led to a lower interferometric small-world coefficient. Thus, for the small-world interferometer model, interferometric measures are necessary to capture the effect of interference on signal transfer. This model is an example of the type of problem that necessitates interferometric measures, and applies to any wave-based network including quantum networks. 

   more » « less
2. Latent Weight-Based Pruning for Small Binary Neural Networks

   https://doi.org/10.1145/3566097.3567873  

   Chen, Tianen; Anderson, Noah; Kim, Younghyun (January 2023, ASPDAC '23: Proceedings of the 28th Asia and South Pacific Design Automation Conference)

   Binary neural networks (BNNs) substitute complex arithmetic operations with simple bit-wise operations. The binarized weights and activations in BNNs can drastically reduce memory requirement and energy consumption, making it attractive for edge ML applications with limited resources. However, the severe memory capacity and energy constraints of low-power edge devices call for further reduction of BNN models beyond binarization. Weight pruning is a proven solution for reducing the size of many neural network (NN) models, but the binary nature of BNN weights make it difficult to identify insignificant weights to remove. In this paper, we present a pruning method based on latent weight with layer-level pruning sensitivity analysis which reduces the over-parameterization of BNNs, allowing for accuracy gains while drastically reducing the model size. Our method advocates for a heuristics that distinguishes weights by their latent weights, a real-valued vector used to compute the pseudogradient during backpropagation. It is tested using three different convolutional NNs on the MNIST, CIFAR-10, and Imagenette datasets with results indicating a 33%--46% reduction in operation count, with no accuracy loss, improving upon previous works in accuracy, model size, and total operation count. 

   more » « less
3. Hierarchical learning and denoising with an olfaction-inspired neuromorphic network

   Moyal, Roy; Einhorn, Matthew; Borthakur, Ayon; Cleland, Thomas Andrew (October 2024, Society for Neuroscience)

   The goal of odor source separation and identification from real-world data presents a challenging problem. Both individual odors of potential interest and multisource odor scenes constitute linear combinations of analytes present at different concentrations. The mixing of these analytes can exert nonlinear and even nonmonotonic effects on cross-responsive chemosensors, effectively occluding diagnostic activity patterns across the array. Neuromorphic algorithms, inspired by specific computational strategies of the mammalian olfactory system, have been trained to rapidly learn and reconstruct arbitrary odor source signatures in the presence of background interference. However, such networks perform best when tuned to the statistics of well-behaved inputs, normalized and predictable in their activity distributions. Deployment of chemosensor arrays in the wild exposes these networks to disruptive effects that exceed these tolerances. To address the problems inherent to chemosensory signal conditioning and representation learning, the olfactory bulb deploys an array of strategies: (1) shunting inhibition in the glomerular layer implements divisive normalization, contributing to concentration-invariant representations; (2) feedforward gain diversification (synaptic weight heterogeneity) regularizes spiking activity in the external plexiform layer (mitral and granule cells), enabling the network to handle unregulated inputs; (3) gamma-band oscillations segment activity into packets, enabling a spike phase code and iterative denoising; (4) excitatory and inhibitory spike timing dependent learning rules induce hierarchical attraction basins, enabling the network to map its highly complex inputs to regions of a lower dimensional manifold; (5) neurogenesis in the granule cell layer enables lifelong learning and prevents order effects (regularizing the learned synaptic weight distribution over the span of training). Here, we integrate these motifs into a single neuromorphic model, bringing together prior OB-inspired model architectures. In a series of simulation experiments including real-world data from a chemosensor array, we demonstrate the network’s ability to learn and detect complex odorants in variable environments despite unpredictable noise distributions. 

   more » « less
4. Continual Learning with Deep Artificial Neurons

   Camp, Brendan; Mandivarapu, Jaya K.; Estrada, Rolando (April 2022, International Conference on Learning Representations (ICLR) Workshop: From Cells to Societies)

   Neurons in real brains are complex computational units, capable of input-specific damping, inter-trial memory, and context-dependent signal processing. Artificial neurons, on the other hand, are usually implemented as simple weighted sums. Here we explore if increasing the computational power of individual neurons can yield more powerful neural networks. Specifically, we introduce Deep Artificial Neurons (DANs)—small neural networks with shared, learnable parameters embedded within a larger network. DANs act as filters between nodes in the net-work; namely, they receive vectorized inputs from multiple neurons in the previous layer, condense these signals into a single output, then send this processed signal to the neurons in the subsequent layer. We demonstrate that it is possible to meta-learn shared parameters for the various DANS in the network in order to facilitate continual and transfer learning during deployment. Specifically, we present experimental results on (1) incremental non-linear regression tasks and (2)unsupervised class-incremental image reconstruction that show that DANs allow a single network to update its synapses (i.e., regular weights) over time with minimal forgetting. Notably, our approach uses standard backpropagation, does not require experience replay, and does need separate wake/sleep phases. 

   more » « less
5. LDPC Decoding With Degree-Specific Neural Message Weights and RCQ Decoding

   https://doi.org/10.1109/TCOMM.2023.3343387  

   Wang, Linfang; Terrill, Caleb; Divsalar, Dariush; Wesel, Richard D (April 2024, IEEE Transactions on Communications)

   Recently, neural networks have improved MinSum message-passing decoders for low-density parity-check (LDPC) codes by multiplying or adding weights to the messages, where the weights are determined by a neural network. The neural network complexity to determine distinct weights for each edge is high, often limiting the application to relatively short LDPC codes. Furthermore, storing separate weights for every edge and every iteration can be a burden for hardware implementations. To reduce neural network complexity and storage requirements, this paper proposes a family of weight-sharing schemes that use the same weight for edges that have the same check node degree and/or variable node degree. Our simulation results show that node-degree-based weight-sharing can deliver the same performance requiring distinct weights for each node. This paper also combines these degree-specific neural weights with a reconstruction-computation-quantization (RCQ) decoder to produce a weighted RCQ (W-RCQ) decoder. The W-RCQ decoder with node-degree-based weight sharing has a reduced hardware requirement compared with the original RCQ decoder. As an additional contribution, this paper identifies and resolves a gradient explosion issue that can arise when training neural LDPC decoders. 

   more » « less