We use a recently developed synchronous Spiking Neural Network (SNN) model
to study the problem of learning hierarchicallystructured concepts. We introduce an abstract data model that describes simple hierarchical concepts. We define a feedforward layered SNN model, with learning modeled using Oja’s local learning rule, a well known biologicallyplausible rule for adjusting synapse weights. We define what it means for such a network to recognize hierarchical concepts; our notion of recognition is robust, in that it tolerates a bounded amount of noise. Then, we present a learning algorithm by which a layered network may learn to recognize hierarchical concepts according to our robust definition. We analyze correctness and performance rigorously; the amount of time required to learn each concept, after learning all of the subconcepts, is approximately O ( 1ηk(`max log(k) + 1ε) + b log(k)), where k is the number of subconcepts per concept, `max is the maximum hierarchical depth, η is the learning rate, ε describes the amount of uncertainty allowed in robust recognition, and b describes the amount of weight decrease for "irrelevant" edges. An interesting feature of this algorithm is that it allows the network to learn subconcepts in a highly interleaved manner. This algorithm assumes that the concepts are presented in a noisefree way; we also extend these results to accommodate noise in the learning process. Finally, we give a simple lower bound saying that, in order to recognize concepts with hierarchical depth two with noisetolerance, a neural network should have at least two layers.
The results in this paper represent first steps in the theoretical study of hierarchical concepts using SNNs. The cases studied here are basic, but they suggest many directions for extensions to more elaborate and realistic cases.
more »
« less
Learning Hierarchically Structured Concepts
We use a recently developed synchronous Spiking Neural Network (SNN) model
to study the problem of learning hierarchicallystructured concepts. We introduce
an abstract data model that describes simple hierarchical concepts. We define a
feedforward layered SNN model, with learning modeled using Oja’s local learning
rule, a well known biologicallyplausible rule for adjusting synapse weights. We
define what it means for such a network to recognize hierarchical concepts; our
notion of recognition is robust, in that it tolerates a bounded amount of noise.
Then, we present a learning algorithm by which a layered network may learn
to recognize hierarchical concepts according to our robust definition. We an
alyze correctness and performance rigorously; the amount of time required to
learn each concept, after learning all of the subconcepts, is approximately
O( 1ηk(`max log(k) + 1ε) + b log(k)), where k is the number of subconcepts
per concept, `max is the maximum hierarchical depth, η is the learning rate, ε
describes the amount of uncertainty allowed in robust recognition, and b describes
the amount of weight decrease for "irrelevant" edges. An interesting feature of this
algorithm is that it allows the network to learn subconcepts in a highly interleaved
manner. This algorithm assumes that the concepts are presented in a noisefree
way; we also extend these results to accommodate noise in the learning process.
Finally, we give a simple lower bound saying that, in order to recognize concepts
with hierarchical depth two with noisetolerance, a neural network should have at
least two layers.
The results in this paper represent first steps in the theoretical study of hierarchical
concepts using SNNs. The cases studied here are basic, but they suggest many
directions for extensions to more elaborate and realistic cases
more »
« less
 Award ID(s):
 2139936
 NSFPAR ID:
 10405675
 Date Published:
 Journal Name:
 Neural networks
 Volume:
 143
 ISSN:
 08936080
 Page Range / eLocation ID:
 798817
 Format(s):
 Medium: X
 Sponsoring Org:
 National Science Foundation
More Like this


We continue our study from [5], of how concepts that have hierarchical structure might be represented in brainlike neural networks, how these representations might be used to recognize the concepts, and how these representations might be learned. In [5], we considered simple treestructured concepts and feedforward layered networks. Here we extend the model in two ways: we allow limited overlap between children of different concepts, and we allow networks to include feedback edges. For these more general cases, we describe and analyze algorithms for recognition and algorithms for learning.more » « less

We continue our study from Lynch and MallmannTrenn (Neural Networks, 2021), of how concepts that have hierarchical structure might be represented in brainlike neural networks, how these representations might be used to recognize the concepts, and how these representations might be learned. In Lynch and MallmannTrenn (Neural Networks, 2021), we considered simple treestructured concepts and feedforward layered networks. Here we extend the model in two ways: we allow limited overlap between children of different concepts, and we allow networks to include feedback edges. For these more general cases, we describe and analyze algorithms for recognition and algorithms for learning.more » « less

Asynchronous eventdriven computation and communication using spikes facilitate the realization of spiking neural networks (SNN) to be massively parallel, extremely energy efficient and highly robust on specialized neuromorphic hardware. However, the lack of a unified robust learning algorithm limits the SNN to shallow networks with low accuracies. Artificial neural networks (ANN), however, have the backpropagation algorithm which can utilize gradient descent to train networks which are locally robust universal function approximators. But backpropagation algorithm is neither biologically plausible nor neuromorphic implementation friendly because it requires: 1) separate backward and forward passes, 2) differentiable neurons, 3) highprecision propagated errors, 4) coherent copy of weight matrices at feedforward weights and the backward pass, and 5) nonlocal weight update. Thus, we propose an approximation of the backpropagation algorithm completely with spiking neurons and extend it to a local weight update rule which resembles a biologically plausible learning rule spiketimingdependent plasticity (STDP). This will enable error propagation through spiking neurons for a more biologically plausible and neuromorphic implementation friendly backpropagation algorithm for SNNs. We test the proposed algorithm on various traditional and nontraditional benchmarks with competitive results.more » « less

Building accurate and efficient deep neural network (DNN) models for intelligent sensing systems to process data locally is essential. Spiking neural networks (SNNs) have gained significant popularity in recent years because they are more biologicalplausible and energyefficient than DNNs. However, SNNs usually have lower accuracy than DNNs. In this paper, we propose to use SNNs for image sensing applications. Moreover, we introduce the DNNSNN knowledge distillation algorithm to reduce the accuracy gap between DNNs and SNNs. Our DNNSNN knowledge distillation improves the accuracy of an SNN by transferring knowledge between a DNN and an SNN. To better transfer the knowledge, our algorithm creates two learning paths from a DNN to an SNN. One path is between the output layer and another path is between the intermediate layer. DNNs use real numbers to propagate information between neurons while SNNs use 1bit spikes. To empower the communication between DNNs and SNNs, we utilize a decoder to decode spikes into real numbers. Also, our algorithm creates a learning path from an SNN to a DNN. This learning path better adapts the DNN to the SNN by allowing the DNN to learn the knowledge from the SNN. Our SNN models are deployed on Loihi, which is a specialized chip for SNN models. On the MNIST dataset, our SNN models trained by the DNNSNN knowledge distillation achieve better accuracy than the SNN models on GPU trained by other training algorithms with much lower energy consumption per image.more » « less