<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Rethinking the performance comparison between SNNS and ANNS</title></titleStmt>
			<publicationStmt>
				<publisher></publisher>
				<date>01/01/2020</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10188417</idno>
					<idno type="doi">10.1016/j.neunet.2019.09.005</idno>
					<title level='j'>Neural Networks</title>
<idno>0893-6080</idno>
<biblScope unit="volume">121</biblScope>
<biblScope unit="issue">C</biblScope>					

					<author>Lei Deng</author><author>Yujie Wu</author><author>Xing Hu</author><author>Ling Liang</author><author>Yufei Ding</author><author>Guoqi Li</author><author>Guangshe Zhao</author><author>Peng Li</author><author>Yuan Xie</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Artificial neural networks (ANNs), a popular path towards artificial intelligence, have experienced remarkable success via mature models, various benchmarks, open-source datasets, and powerful computing platforms. Spiking neural networks (SNNs), a category of promising models to mimic the neuronal dynamics of the brain, have gained much attention for brain inspired computing and been widely deployed on neuromorphic devices. However, for a long time, there are ongoing debates and skepticisms about the value of SNNs in practical applications. Except for the low power attribute benefit from the spike-driven processing, SNNs usually perform worse than ANNs especially in terms of the application accuracy. Recently, researchers attempt to address this issue by borrowing learning methodologies from ANNs, such as backpropagation, to train high-accuracy SNN models. The rapid progress in this domain continuously produces amazing results with ever-increasing network size, whose growing path seems similar to the development of deep learning. Although these ways endow SNNs the capability to approach the accuracy of ANNs, the natural superiorities of SNNs and the way to outperform ANNs are potentially lost due to the use of ANN-oriented workloads and simplistic evaluation metrics.In this paper, we take the visual recognition task as a case study to answer the questions of ''what workloads are ideal for SNNs and how to evaluate SNNs makes sense''. We design a series of contrast tests using different types of datasets (ANN-oriented and SNN-oriented), diverse processing models, signal conversion methods, and learning algorithms. We propose comprehensive metrics on the application accuracy and the cost of memory & compute to evaluate these models, and conduct extensive experiments. We evidence the fact that on ANN-oriented workloads, SNNs fail to beat their ANN counterparts; while on SNN-oriented workloads, SNNs can fully perform better. We further demonstrate that in SNNs there exists a trade-off between the application accuracy and the execution cost, which will be affected by the simulation time window and firing threshold. Based on these abundant analyses, we recommend the most suitable model for each scenario. To the best of our knowledge, this is the first work using systematical comparisons to explicitly reveal that the straightforward workload porting from ANNs to SNNs is unwise although many works are doing so and a comprehensive evaluation indeed matters. Finally, we highlight the urgent need to build a benchmarking framework for SNNs with broader tasks, datasets, and metrics.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Introduction</head><p>Artificial neural networks (ANNs) <ref type="bibr">(LeCun, Bengio, &amp; Hinton, 2015)</ref> are able to learn high-level features from large amount of <ref type="bibr">(Graves et al., 2016)</ref>, etc., and sometimes the combination of CNNs and RNNs <ref type="bibr">(Caglayan &amp; Burak Can, 2018;</ref><ref type="bibr">Zhang, Bai, &amp; Zhu, 2019;</ref><ref type="bibr">Zoph, Vasudevan, Shlens, &amp; Le, 2018)</ref>. Besides various models and learning algorithms, the big data resources (e.g. ImageNet dataset <ref type="bibr">(Deng et al., 2009)</ref> for image recognition) and highperformance computing platforms (e.g. GPU) further boost the development of ANNs. The above successes motivate numerous researches on ANN-specific accelerators <ref type="bibr">(Chen et al., 2014;</ref><ref type="bibr">Chen, Krishna, Emer, &amp; Sze, 2017;</ref><ref type="bibr">Jouppi et al., 2017;</ref><ref type="bibr">Yin et al., 2017)</ref>.</p><p>Spiking neural networks (SNNs) <ref type="bibr">(Ghosh-Dastidar &amp; Adeli, 2009;</ref><ref type="bibr">Maass, 1997)</ref> closely mimic the behaviors of biological neural circuits. They operate with continuous spatio-temporal dynamics and event-driven firing activities (0-nothing or 1-spike event). Due to the asynchronous spiking mechanism, SNNs have shown advantages in event-based scenarios such as optical flow estimation <ref type="bibr">(Haessig, Cassidy, Alvarez, Benosman, &amp; Orchard, 2017)</ref>, spike pattern recognition <ref type="bibr">(Wu et al., 2019)</ref>, and SLAM <ref type="bibr">(Vidal, Rebecq, Horstschaefer, &amp; Scaramuzza, 2018)</ref>. Besides, they are also promising in addressing some interesting problems, for instance, probabilistic inference <ref type="bibr">(Maass, 2014)</ref>, heuristically solving NP-hard problem <ref type="bibr">(Jonke, Habenschuss, &amp; Maass, 2016)</ref> or quickly solving optimization problem <ref type="bibr">(Davies et al., 2018)</ref>, sparse representation <ref type="bibr">(Shi, Liu, Wang, Li, &amp; Gu, 2017)</ref>, and robotics <ref type="bibr">(Hwu, Isbell, Oros, &amp; Krichmar, 2017)</ref>. Furthermore, SNNs are widely deployed in neuromorphic devices for brain inspired computing <ref type="bibr">(Davies et al., 2018;</ref><ref type="bibr">Furber, Galluppi, Temple, &amp; Plana, 2014;</ref><ref type="bibr">Merolla et al., 2014;</ref><ref type="bibr">Shi et al., 2015)</ref>. Whereas, there are ongoing debates for a long time about the practical value of SNNs as a computational tool in both AI and neuromorphic computing communities <ref type="bibr">(Davies et al., 2018)</ref>, especially when compared to ANNs. These skepticisms slow down the development of neuromorphic computing during the past few years, which is upstaged by the rapid progress of deep learning. Researchers attempt to fundamentally mitigate this issue by strengthening SNNs from means such as training algorithm design.</p><p>Unlike the mature and effective training algorithms such as error backpropagation (BP) for ANNs, one of the most difficulties in SNN study is the hardness of training caused by the complex dynamics and non-differentiable spike activities. The case still remains challenging even if we use the simple leakyintegrate-and-fire (LIF) <ref type="bibr">(Gerstner, Kistler, Naud, &amp; Paninski, 2014)</ref> neuron model. To enhance the application accuracy of SNNs, the conventional spike timing dependent plasticity (STDP) unsupervised learning rule can be improved such as by adding lateral inhibition and adaptive threshold <ref type="bibr">(Diehl &amp; Cook, 2015)</ref> or reward mechanism <ref type="bibr">(Mozafari, Ganjtabesh, Nowzari-Dalini, Thorpe, &amp; Masquelier, 2018)</ref>. Some other works pre-train adapted ANNs and convert them to SNNs <ref type="bibr">(Cao, Chen, &amp; Khosla, 2015;</ref><ref type="bibr">Diehl et al., 2015;</ref><ref type="bibr">Hu, Tang, Wang, &amp; Pan, 2018;</ref><ref type="bibr">Hunsberger &amp; Eliasmith, 2016;</ref><ref type="bibr">Rueckauer, Lungu, Hu, Pfeiffer, &amp; Liu, 2017;</ref><ref type="bibr">Sengupta, Ye, Wang, Liu, &amp; Roy, 2019)</ref>. The adaptions in ANNs usually include removing biases, using ReLU activation function (or its variant), changing max pooling to average pooling, etc., to enhance the compatibility with SNN models. The conversion from ANNs to SNNs often induces weight/activation normalization, threshold tuning, sampling error compensation, etc., to maintain the accuracy. Recently, the supervised BP learning in ANNs is borrowed to train accurate SNNs directly <ref type="bibr">(Jin, Li, &amp; Zhang, 2018;</ref><ref type="bibr">Lee, Delbruck, &amp; Pfeiffer, 2016;</ref><ref type="bibr">Pengjie Gu &amp; Tang, 2019;</ref><ref type="bibr">Shrestha &amp; Orchard, 2018;</ref><ref type="bibr">Wu, Deng, Li, Zhu, &amp; Shi, 2018;</ref><ref type="bibr">Wu et al., 2019)</ref>. The gradients can propagate along only the spatial direction by aggregating spikes in the temporal dimension when performing BP, or propagate along both temporal and spatial dimensions by directly calculating the derivates of membrane potential and spike activity at each time step. The combination of BP and STDP learning also exists, such as applying STDP update after the BP update at each training iteration <ref type="bibr">(Tavanaei &amp; Maida, 2017)</ref> or applying BP fine-tuning after an STDP pre-training <ref type="bibr">(Lee, Panda, Srinivasan, &amp; Roy, 2018)</ref>. In a nutshell, with the above efforts, SNNs are gradually approaching the ANN-level application accuracy in visual recognition tasks.</p><p>Due to the lack of specialized benchmark workloads for SNNs, plenty of works directly port the testing workloads from the ANN domain to verify SNN models <ref type="bibr">(Cao et al., 2015;</ref><ref type="bibr">Diehl &amp; Cook, 2015;</ref><ref type="bibr">Diehl et al., 2015;</ref><ref type="bibr">Hu et al., 2018;</ref><ref type="bibr">Hunsberger &amp; Eliasmith, 2016;</ref><ref type="bibr">Jin et al., 2018;</ref><ref type="bibr">Lee et al., 2016</ref><ref type="bibr">Lee et al., , 2018;;</ref><ref type="bibr">Mozafari et al., 2018;</ref><ref type="bibr">Rueckauer et al., 2017;</ref><ref type="bibr">Sengupta et al., 2019;</ref><ref type="bibr">Tavanaei &amp; Maida, 2017;</ref><ref type="bibr">Wu et al., 2018)</ref>. For example, the image datasets for ANN validation are simply converted to the spike version for SNN training and testing. This seems reasonable because the input data are encoded in spikes when SNNs run. Whereas, the original datasets for ANNs are just static images which cannot make full use of the spatio-temporal superiorities of SNNs even if they are converted to spike patterns. Furthermore, the application accuracy is still the prime evaluation metric, but it is known that our brain usually performs worse than current AI machines in terms of absolute recognition accuracy. This reflects that we need more comprehensive and fair metrics to evaluate the brain inspired SNNs. In a nutshell, due to the inappropriate workloads and evaluation metrics, current SNNs fail to beat ANNs. Therefore, there comes up two open questions that ''what workloads are ideal for SNNs and how to evaluate SNNs makes sense''.</p><p>In this paper, we try to answer the above questions and take the visual recognition task as a case study. In <ref type="bibr">Hunsberger and Eliasmith (2016)</ref> that converts pre-trained ANNs with soft-LIF neurons and noisy activities to SNNs, the efficiencies of SNNs and ANNs are defined and compared. It is one of the few works that consider the metric beyond accuracy in the SNN algorithm community and study the trade-off between accuracy and efficiency. We extend this path to a deeper rethinking via more systematical modeling methodologies and experimental analyses. Specifically, through designing a series of contrast tests using datasets in different domains, diverse processing models, signal conversion methods, and learning algorithms, we compare the performance of ANNs and SNNs (with rate coding). We propose comprehensive metrics on the application accuracy, memory cost, and compute cost to evaluate these models and conduct extensive analyses. We evidence the fact that SNNs fail to beat ANNs at the same network scale on ANN-oriented workloads with respect to the accuracy but hold the potential for efficient processing; while on SNNoriented workloads, SNNs can fully perform better. We further demonstrate that the change of simulation time window and firing threshold will produce a trade-off between the application accuracy and the execution cost. Based on these analyses, we recommend the most suitable model for each scenario. The major contributions of this work are summarized as follows:</p><p>&#8226; We propose the much-needed comprehensive evaluation metrics for SNNs and ANNs by taking the trade-off between application accuracy and execution cost into account. We conduct extensive experiments and analyses with diverse benchmark datasets, processing models, signal conversions, and learning algorithms. Various comparison methodologies and visualization means are also demonstrated.</p><p>&#8226; Based on the above results, we recommend the best model for each workload, which provides insightful modeling guidances.</p><p>&#8226; We point out that directly porting workloads from ANNs to SNNs is inappropriate, at least unwise, although many works are doing so. We further highlight that to create more SNNoriented datasets and build a benchmarking framework with broader tasks is urgent for the SNN community. This shall shed light on the future opportunities of SNN research. The rest of this paper is organized as follows: Section 2 introduces some preliminaries of ANNs and SNNs, as well as typical network topologies and benchmark datasets; Section 3 systematically proposes signal conversion methods, testing workloads, training algorithms, and evaluation metrics; The experimental setup and result analyses along with visualizations are provided in Section 4; Finally, Section 5 concludes and discusses the paper.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Preliminaries</head><p>In this section, we give the preliminary knowledge of ANNs and SNNs for visual recognition, including neuron models, network topologies, and benchmark datasets.</p><p>Neurons are the basic compute units, which are wired by abundant synapses to form a neural network. If we treat the neural network as a graph, each neuron and synapse can be viewed as a node and edge, respectively. As aforementioned, there are two categories of neural networks: ANNs and SNNs, which will be explained in detail as follows.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.1.">Artificial neural networks</head><p>Fig. <ref type="figure">1</ref>(a) depicts the model of a typical artificial neuron. The compute process is governed by</p><p>where x, y, w, and b are input activation, output activation, synaptic weight, and bias, respectively, and j is the index of input neuron. &#981;(&#8226;) is a nonlinear activation function, e.g. &#981;(x) = ReLU(x) = max(x, 0). Neurons in ANNs communicate with each other using activations coded in high-precision and continuous values, and only propagate information in the spatial domain (i.e. layer by layer). From the above equation, it can be seen that the multiply-and-accumulate (MAC) of inputs and weights is the major operation in ANNs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2.">Spiking neural networks</head><p>Fig. <ref type="figure">1</ref>(b) shows a typical spiking neuron, which has a similar structure but different behavior compared to the ANN neuron. By contrast, spiking neurons communicate through spike trains coded in binary events rather than the continuous activations in ANNs. The dendrites integrate the input spikes and the soma consequently conducts nonlinear transformation to produce the output spike train. This behavior is usually modeled by the popular LIF model <ref type="bibr">(Gerstner et al., 2014)</ref>, described as</p><p>where (t) denotes the time step, &#964; is a time constant, and u and s are the membrane potential and output spike, respectively. u r 1 and u r 2 are the resting potential and reset potential, respectively.</p><p>w j is the synaptic weight from the jth input neuron, t k j is the time when the kth spike of the jth input neuron fires within the integration time window of T w (a spike sequence of S Tw j in total), and K (&#8226;) is a kernel function describing the time decay effect. u th is the firing threshold that determines whether to fire a spike or not. Besides the LIF model, there also exist other neuron models in SNNs, such as the model of <ref type="bibr">Hodgkin and Huxley (1952)</ref> or <ref type="bibr">Izhikevich (2003)</ref>. However, due to the higher complexity they are not widely used in practical SNN models.</p><p>Different from ANNs, SNNs represent information in spike patterns and each spiking neuron experiences rich dynamic behaviors. Specifically, besides the information propagation in the spatial domain, the current state is tightly affected by the past history in the temporal domain. Therefore, SNNs usually have more temporal versatility but lower precision compared to ANNs mainly with spatial propagation and continuous activations. Since a spike only fires when the membrane potential exceeds a threshold, the entire spike signals are often sparse and the compute can be event driven (only enabled when a spike input arrives). Furthermore, because the spike is binary, i.e. 0 or 1, the costly multiplication between the input and weight can be removed if the integration time window T w equals to 1 (see Section 3.5). For above reasons, SNNs can usually achieve lower power consumption compared to ANNs with intensive computation.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.3.">Typical network topologies</head><p>The basic layer topologies used to build neural networks are fully-connected (FC) layer, recurrent layer, and convolutional (Conv) layer (as well as pooling layer). The corresponding networks are named as multi-layered perceptron (MLP), RNNs, and CNNs, respectively. MLP and RNNs only include stacked FC layers with or without recurrent connections in each layer as shown in Fig. <ref type="figure">2</ref>(a) and (b), respectively. For CNNs shown in Fig. <ref type="figure">2(c)</ref>, they directly target the processing of 2D features instead of the 1D ones in MLP and RNNs. Each neuron in the convolutional (Conv) layer only receives inputs from a local receptive field (RF) across all feature maps (FMs) in previous layer. The basic calculation of each neuron is the same as Eq. ( <ref type="formula">1</ref>) or (2). In addition, CNNs also use the pooling layer to down sample the size of each FM individually via outputting the maximum (i.e. max pooling) or average (i.e. average pooling) value of each RF, and use the FC layer for final classification.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.4.">Benchmark datasets</head><p>Fig. <ref type="figure">3</ref> illustrates two different types of benchmark datasets we used for visual recognition. The upper two rows are sampled from <ref type="bibr">MNIST (LeCun, Bottou, Bengio, &amp; Haffner, 1998</ref>) and CIFAR10 <ref type="bibr">(Krizhevsky &amp; Hinton, 2009)</ref> datasets. Because they are frame-based static images and widely used in ANNs, we call  them ANN-oriented datasets. The lower two rows are sampled from N-MNIST <ref type="bibr">(Orchard, Jayawant, Cohen, &amp; Thakor, 2015)</ref> and DVS-CIFAR10 <ref type="bibr">(Li, Liu, Ji, Li, &amp; Shi, 2017)</ref>. The data format is spike event which is converted from the above static datasets through scanning each image by using dynamic vision sensor (DVS) <ref type="bibr">(Lichtsteiner, Posch, &amp; Delbruck, 2008)</ref>. Besides the similar spatial information as ANN-oriented datasets, it contains more dynamic temporal information and the spike events are naturally compatible with the signal format in SNNs, therefore we call them SNN-oriented datasets.</p><p>For the ANN-oriented datasets, MNIST is comprised of a training set with 60,000 labeled hand-written digits, and a testing set of other 10,000 labeled digits. Each digit sample is a 28 &#215; 28 grayscale image. CIFAR10 contains a training set with 50,000 labeled training images with natural and manufactured objects in their environment, and other 10,000 testing images. Each sample is a 32 &#215; 32 &#215; 3 RGB image.</p><p>For the SNN-oriented datasets, DVS can scan the original static image along given directions and collect the generated spike trains triggered by the intensity change of each pixel. Since it has two change manners (increase or decrease), DVS produces two channels of spike events, named as On and Off event (red and blue color in Fig. <ref type="figure">3</ref>, respectively). Hence, DVS converts each image into row &#215; col &#215; 2 &#215; T spike pattern, where T is the recording time length. Due to the relative shift during the movement, the resulting images usually have a larger size than original images. In this way, N-MNIST converts the original MNIST into its spike version. It also contains 60,000 training samples and 10 000 testing samples and each sample in N-MNIST is a spatio-temporal </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Benchmarking methodology</head><p>In this section, we first introduce the signal conversion between the ANN and SNN domains. Then we design six benchmark models along with the training algorithms for contrast tests and propose three evaluation metrics for consequent model assessment and comparison.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1.">Data signal conversion</head><p>In general, ANNs receive frame-based images in real value while SNNs receive event-driven spike signals. Thus, sometimes it is necessary to transform the same data resource into a different form for the processing in the other domain. Here we take the visual recognition task as a case study and primarily introduce four signal conversion methods as follows.</p><p>Image to spike pattern. Since real-valued signals of pixel intensity are not suitable for spike-based SNNs, the conversion to spike trains is needed when testing SNN models over ANNoriented datasets. One of the prevalent strategies is the probabilistic sampling. At every time step, it samples the original pixel intensity (usually normalized to [0, 1]) into a binary value, wherein the probability of being 1 (firing a spike) equals to the intensity value. The sampling follows a specific probability distribution such as Bernoulli distribution or Poisson distribution. For example, the i 1 neuron in Fig. <ref type="figure">4(a)</ref>, corresponding to the pixel on the upper left with a normalized intensity of 0.8, produces a binary spike train following the Bernoulli distribution B(0.8, T ). Here T is a given time window for sampling.</p><p>The above element-wise sampling usually suffers from precision loss in the case of short time window (see Section 4.2). To circumvent this problem, modern works <ref type="bibr">(Esser et al., 2016)</ref> add an encoding layer to generate spike signals globally, as shown in <ref type="bibr">Fig. 4(b)</ref>. Each neuron in this layer receives intensity values of multiple pixels as the input (i.e. the dendrite works in ANN mode with normal input-weight MAC operations) while produces spikes as the output (i.e. the soma works in SNN mode with LIF dynamics). Although the encoding layer is an ANN-SNN hybrid layer rather than a complete SNN layer like following layers in the network, its weights are trainable because our training method for SNNs in this paper is also BP-compatible for ANNs (see Section 3.4). Since the number of neurons can be flexibly customized and the parameters are trainable, it can adapt to the overall optimization problem for better accuracy.</p><p>Spike pattern to image. Similarly, to test ANN models over SNN-oriented datasets, we need to convert spike patterns into frame-based images. There exist two possible output formats: (i) binary image with 0/1 pixels; (ii) intensity image with realvalued pixels. An intuitive strategy for converting a spike pattern into binary images is to first expand the spike trains from all neuron locations along the temporal dimension. Then, as Fig. <ref type="figure">4(c</ref>) illustrates, the expanded 2D spike pattern (location index v.s. time) can be directly viewed as a binary image (every spike event represents the pixel intensity being 1, otherwise the pixel intensity is 0). The original recording time length T is usually long which might result in a too big image size. So the spike events along the temporal direction need to be aggregated or sampled periodically with a moderate sliding time window (e.g. every 1 ms). Here the aggregation means the resulting spike event is 1 if it has spikes in the window, otherwise it is 0. Even if so, the unfolding of 2D locations (the left side in Fig. <ref type="figure">4(c)</ref>) to an 1D location vector (Y-axis of the right side in Fig. <ref type="figure">4(c</ref>)) also produces a big image size.</p><p>For the conversion into intensity images, a temporal accumulation of the spike events (counting spike numbers) is required. Fig. <ref type="figure">4</ref>(d) depicts the accumulation process of spike trains within 100 ms. The accumulated spike numbers will be normalized to be pixels with a proper intensity value. Due to the relative motion and intrinsic noise of DVS, the resulting image is often blurred and its edge features are obscure. This conversion is only allowed by a strong assumption that each spike location should not move away its beginning location as t evolves, otherwise it will severely harm the quality of resulting image.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2.">ANN-oriented workloads</head><p>Here the ''ANN-oriented workloads'' means the target is to recognize the images in frame-based datasets (e.g. MNIST and CIFAR10), which is widely used in the ANN field. To process this type of workloads, we introduce three benchmark models. As shown in Fig. <ref type="figure">5</ref>(a), the most straightforward solution is the natural ANN with ''ANN training &amp; ANN inference''. It fully trains the network in the ANN mode (Eq. ( <ref type="formula">1</ref>)) with intensity images and then does consequent inference in the same domain. The training follows the most widely used BP algorithm in the ANN field.</p><p>Besides, many works in the SNN community also use these datasets to test the performance of SNNs <ref type="bibr">(Diehl et al., 2015;</ref><ref type="bibr">Hu et al., 2018;</ref><ref type="bibr">Lee et al., 2016;</ref><ref type="bibr">Sengupta et al., 2019;</ref><ref type="bibr">Wu et al., 2018</ref><ref type="bibr">Wu et al., , 2019))</ref>. Since in these cases the network works in the SNN mode, the input data need to be converted from frame-based images to spike events, as illustrated in Fig. <ref type="figure">4</ref>(a) or (b). After the signal conversion, we further provide two modeling branches. One is shown in <ref type="bibr">Fig. 5(b)</ref>, where an ANN is first trained using BP algorithm on the original image dataset, and then the pre-trained ANN is adapted to its SNN counterpart with the same structure but different neuron model. This converted SNN receives the dataset variant with spike events in the inference phase. The other is the enforced SNN as depicted in Fig. <ref type="figure">5(c)</ref>, where an SNN model is directly trained from scratch on the converted spike datasets and tested in the same domain. As aforementioned in Section 1, the emerging BP-inspired supervised algorithms for training SNNs usually present better accuracy than the unsupervised ones. Actually they use different learning methodologies. For example, both synaptic weights and axonal delays are trainable variables under the error backpropagation framework in <ref type="bibr">Shrestha and Orchard (2018)</ref>, the gradients at every location and time are directly  derived using spatio-temporal backpropagation (STBP) method in <ref type="bibr">Wu et al. (2018)</ref> and <ref type="bibr">Wu et al. (2019)</ref>, while a temporal based loss function is further proposed to solve the spatio-temporal credit assignment problem in Pengjie <ref type="bibr">Gu and Tang (2019)</ref>. Considering the open-source codes in Pytorch, we select the STBP for the direct training of SNNs. In summary, the three benchmark models for ANN-oriented workloads are denoted as Model-1/2/3 for clarity. Note that we use the rate coding scheme in the SNN domain throughout this paper, which is because of its higher accuracy indicated by previous work.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3.">SNN-oriented workloads</head><p>Here the ''SNN-oriented workloads'' means the target is to recognize the images in frame-free spike datasets (e.g. N-MNIST and DVS-CIFAR10), which is widely used in the SNN field. Like the above ANN-oriented workloads, here we also introduce three benchmark models. The first two work in the ANN mode (see Fig. <ref type="figure">6(a)</ref>), where the input data are converted into images. The models are trained using the regular BP algorithm for ANNs and tested also in the ANN domain. There are two ways for the data conversion from spike events to images: (i) binary images by directly receiving the unfolded spike pattern; (ii) intensity images by compressing the spike pattern along the temporal dimension, which are illustrated in Fig. <ref type="figure">4(c</ref>) and (d), respectively. Fig. <ref type="figure">6</ref> gives another benchmark model that works in the natural SNN mode. The network is directly trained and tested using the original spike dataset, and the learning algorithm is still BP-inspired as above in Model-3. Similarly, the three benchmark models for SNN-oriented workloads are denoted as Model-4/5/6.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.4.">Training algorithms</head><p>In the above section, we mentioned two training algorithms for the six benchmark models: (i) BP for ANNs; (ii) BP-inspired variant for SNNs. The BP algorithm is widely used in the training of ANN. It can be simply described as</p><p>where i and j are the neuron index in previous layer n and current layer (n+1), respectively, &#981;</p><p>is the activation derivative of the jth neuron in layer (n + 1), and L is the loss function for optimization. Usually, we can simply use mean square error (MSE) as the cost to be minimized, i.e. let</p><p>Here Y Y Y and Y Y Y label are the actual output and labeled ground truth, respectively.</p><p>As aforementioned, because of the superior accuracy and open-source codes, we select STBP <ref type="bibr">(Wu et al., 2018</ref><ref type="bibr">(Wu et al., , 2019) )</ref> to train our SNNs. It is based on an iterative version of the original LIF model in Eq. ( <ref type="formula">2</ref>). Specifically, it yields</p><p>where dt is the length of simulation time step, o denotes the neuronal spike output, t and n are time step and layer index, respectively. e -dt &#964; reflects the decay effect of the membrane potential. f (&#8226;) is a step function, which satisfies f (x) = 1 when x &#8805; 0, otherwise f (x) = 0. This iterative LIF format incorporates all the behaviors in the original neuron model including integration, firing, and reset. Note that here we set u r 1 = u r 2 = 0, T w = 1, and K (&#8226;) &#8801; 1 in the original LIF model for simplicity. Given the iterative LIF model, the gradient propagates along both the temporal and spatial dimensions, and the parameter update can be derived accordingly as follows</p><p>(5)</p><p>To solve the non-differentiable problem, <ref type="bibr">Wu et al. (2018)</ref> introduces an auxiliary function to approximate the derivative of step function f (&#8226;) when calculating &#8706;o &#8706;u . It satisfies</p><p>where the parameter a determines the gradient width. We have sign(x) = 1 when x &gt; 0 and sign(x) = 0 when x = 0. The loss function L measures the discrepancy between the ground truth and the average firing rate of the last layer N within a given time window T . Also, it can be defined as the format of MSE like that in ANNs, i.e.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.5.">Evaluation metrics</head><p>As well known, the SNNs-based brain usually cannot beat the current ANNs-based AI system in terms of the absolute recognition accuracy, while the true brain performs better on other metrics, such as operating efficiency. Whereas, in most recent works, recognition accuracy is still a mainstream metric to judge which model (ANNs or SNNs) is better, especially in the algorithm research. This is not fair enough since ANNs and SNNs have very different characteristics. For example, the data precision in ANNs are much higher than SNNs, which makes ANNs easier to gain better recognition accuracy at the same network size. All these indicate more comprehensive metrics are needed for model evaluation. Besides the common accuracy comparison, here we further introduce memory and compute cost as complementary evaluation metrics. Note that this is just a beginning, and more insightful metrics are expected in the future (see discussions in Section 5.2).</p><p>Recognition accuracy. Here we use the top-1 accuracy. In ANNs, this accuracy means the percentage of the correctly recognized samples. If the label class is the same with the one predicted by the model with the maximum activation value, the recognition is correct for the current sample. In SNNs, we first count the fire rate (number of spikes) of the output neurons during a given time window T for rate coding, then determine the predicted class &#264; according to the fire rate values. The consequent accuracy calculation is the same as that in ANNs. A well-accepted decoding method can be defined as follows</p><p>where o t,N i denotes the spike output of the ith neuron in the last layer N at the tth time step.</p><p>Notation: Before stepping into the introduction of memory and compute costs, we make a clarification that in this paper we consider costs in only the inference phase. On one hand, the training of SNNs under Eq. ( <ref type="formula">5</ref>) with spatio-temporal gradient propagation paths is more complicated than that of the inference phase. The complication will weaken the readability of this work since it is already complicated due to various signal conversions, modeling methods, workloads, and evaluation metrics. On the other hand, most neuromorphic devices supporting SNNs perform only the inference phase. Therefore, it is better to focus on the inference phase that can be quickly applied in current hardware platforms.</p><p>Memory cost. Usually, the memory footprint matters a lot when deploying models on embedded devices. In ANNs, the memory cost includes the weight memory and activation memory. The overhead of the activation function is ignored but it should be counted if it is realized using look up table. In SNNs, the memory cost includes the weight memory, membrane potential memory, and spike memory. Other parameters such as the fire threshold u th and time constant &#964; are negligible since they can be shared by all neurons in the same layer or in the whole network. The spike memory overhead only occurs when spike fires. In summary, the memory cost can be calculated by</p><p>where M w , M a , M p , and M s denote the memory cost for weights, activations, membrane potentials, and spikes, respectively. Compared to the static values of M w , M a , and M p that are determined by the network structure, M s is dynamically determined by the maximum number of spike events at a certain time step. Here for clarity, all the data are assumed to be stored in 32 bits except for the 1-bit spike data. Note that, in models with data quantization <ref type="bibr">(Banner, Hubara, Hoffer, &amp; Soudry, 2018;</ref><ref type="bibr">Deng, Jiao, Pei, Wu, &amp; Li, 2018)</ref> or on specialized devices <ref type="bibr">(Davies et al., 2018;</ref><ref type="bibr">Jouppi et al., 2017;</ref><ref type="bibr">Merolla et al., 2014)</ref>, the bitwidth can be reduced to save the memory cost. For some compressed models via matrix/tensor decomposition <ref type="bibr">(Novikov, Podoprikhin, Osokin, &amp; Vetrov, 2015)</ref> or sparsification <ref type="bibr">(Li, Kadav, Durdanovic, Samet, &amp; Graf, 2016;</ref><ref type="bibr">Liu et al., 2018)</ref> techniques, the memory cost can also be reduced. However, these are out of the scope of this paper. Compute cost. The compute overhead is critical for the running latency and energy consumption. In ANNs, the compute cost is mainly determined by the MAC operations in Eq. ( <ref type="formula">1</ref>), which is widely used in ANN accelerators <ref type="bibr">(Chen et al., 2014</ref><ref type="bibr">(Chen et al., , 2017))</ref>. In SNNs, the major compute overhead is the spike-driven input integration in Eq. ( <ref type="formula">2</ref>). Two differences from ANNs should be emphasized: (i) the costly multiplication can be removed due to the binary spike inputs if T w = 1 and K (&#8226;) &#8801; 1 actually we satisfy these for simplicity, then the dendrite integration becomes multiplierless as</p><p>is event-driven that implies no computation occurs if no spike received. The compute cost can be calculated by</p><p>where C mul and C add are the compute cost for multiplications and additions, respectively. The compute overheads in soma (such as the activation function in ANNs and the membrane potential update and firing activity in SNNs) are ignored, which is a common way in neural network devices <ref type="bibr">(Akopyan et al., 2015;</ref><ref type="bibr">Jouppi et al., 2017)</ref>. Note that in SNNs, the C add takes all the addition operations into account during the entire rate coding period (e.g. T ), which is proportional to the total number of spike events. Similar to the memory cost evaluation, the data quantization can replace the complex high-precision MACs to efficient low-precision ones and the compression techniques (e.g. decomposition and sparsification) can additionally reduce the amount of operations. Whereas, they are not the focus of this paper as a starting point to establish a comprehensive evaluation framework.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.">Experimental results</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.1.">Experimental setup</head><p>In our experiments, we comprehensively evaluate the performance of several ANN and SNN models for visual recognition over different types of benchmark workloads (ANN-oriented and SNNoriented). As a starting work, we discuss more on the results of MLP and (plain) CNNs for simplicity and just provide the results of other models (e.g. RNNs and temporal CNNs) on SNNoriented workloads at the end. On ANN-oriented workloads, we evaluate the performance of Model-1/2/3 on MNIST and CIFAR10 datasets. Due to the incapability of MLP in processing larger-scale CIFAR10, we just show the CNN results on this dataset. On SNNoriented workloads, we evaluate the performance of Model-4/5/6 on the neuromorphic version of above datasets, i.e. N-MNIST and DVS-CIFAR10. Note that why we mainly compare the results across Model 1-6 implemented by ourselves is because we can easily control many factors (e.g. network structure and size, training techniques, learning hyper-parameters, etc.) to guarantee the fairness. Otherwise, it will be challenging to present a fair comparison with others due to the modeling diversity in this field.</p><p>The overall experimental flowchart is presented in Fig. <ref type="figure">7</ref>, and the main network structures are given in Table <ref type="table">1</ref>. Besides,   the parameter configurations for ANN-oriented and SNN-oriented workloads are provided in Tables <ref type="table">2</ref> and<ref type="table">3</ref>, respectively. Since these workloads usually have different encoding formats for input conversion, we just show the parameter configurations of hidden layers for clarity. In all Conv layers, we set the padding value to 1. Due to the large input size of Model-4, we set the stride value to 2 there, and set the stride value to 1 in other models. In all SNN models (Model-2/3/6), each unit in the pooling layer is an independent neuron, which guarantees that the outputs of pooling layer are still in the spike format. This pooling configuration is different from previous work <ref type="bibr">(Wu et al., 2018</ref><ref type="bibr">(Wu et al., , 2019))</ref>. In Model 3/6 we use the complete LIF neuron model described in Eq. ( <ref type="formula">4</ref>), while in Model-2 we use the IF neuron model (without the leakage item in LIF) that is a popular choice in previous converted SNNs <ref type="bibr">(Diehl et al., 2015;</ref><ref type="bibr">Sengupta et al., 2019)</ref>. We implement all the models in Pytorch framework. On MNIST and N-MNIST datasets, we adopt Adam (adaptive moment estimation <ref type="bibr">(Kingma &amp; Ba, 2014)</ref>) optimizer with default parameter setting (&#945; = 0.0001, &#946; 1 = 0.9, &#946; 2 = 0.999, &#1013; = 10 -8 ); while on CIFAR10 and DVS-CIFAR10 datasets, we use SGD (stochastic gradient descent) optimizer with initial learning rate r = 0.1 and momentum 0.9, wherein r decays by 10x every 35 training epochs.   a Refers to the model using probabilistic sampling for input signal conversion.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.2.">Accuracy analysis</head><p>b Refers to the model using encoding layer for input signal conversion.</p><p>based on pre-trained ANNs (Model-2) are comparable but slightly worse than those of natural ANNs. The enforced SNNs (Model-3) trained by BP-inspired algorithm on the converted spike datasets achieve the worst accuracy. Note that although we also observe that the enforced SNN can occasionally surpass the converted SNN (e.g. on the CNN structure), it has never been better than the natural ANN.</p><p>The small gap between different models on MNIST might be due to the reason that it is a simple task. To this end, we continue to test on a larger dataset, i.e. CIFAR10. In this case, the accuracy differences become more obvious, which follows 'natural ANN (Model-1) &#8658; converted SNN (Model-2) &#8658; enforced SNN (Model-3)' from the best to the worst (i.e. 78.16% &#8658; 76.81% &#8658; 63.19%).</p><p>Even though the accuracy of enforced SNN can be improved to 74.23% by using an extra encoding layer (Fig. <ref type="figure">4(b)</ref>) for signal conversion, it still cannot beat the natural ANN.</p><p>From Table <ref type="table">4</ref>, you can find evidence that the converted and enforced SNNs cannot beat natural ANNs. Now, we try to reveal the underlying reason for this accuracy gap. Fig. <ref type="figure">8</ref>(a) presents the information loss when converting the original image to spike signal for consequent SNN processing (in Model-2 &amp; Model-3).</p><p>Here we take the probabilistic sampling in Fig. <ref type="figure">4</ref>(a) as an example. We first count the total number of spikes produced by probabilistic sampling of each pixel during the simulation time window T , and then visualize the spike numbers after being normalized as images. As T increases, the images become clearer which indicates less precision loss in signal conversion. However, a too large T will cause long simulation time. In our experiments, we set T = 15, however, the precision loss for signal conversion still exists. Fig. <ref type="figure">8</ref>(b) depicts the mean absolute errors (MAE) between the original pixel value and normalized spike number as well as the MAE variances due to the nondeterministic sampling. Finally, we study how T affects the training convergence, which is shown by the training curves under different T configuration. Usually, the longer T produces a faster convergence speed and better final accuracy, which is mainly due to the smaller precision loss during input signal conversion. In summary, the signal conversion from images to spikes causes information loss and makes the SNN models lose the chance to outperform the natural ANN on ANNoriented workloads. Furthermore, the length of simulation time window T significantly affects the model convergence.</p><p>Next, let us analyze why the converted SNN also loses accuracy compared to the natural ANN. The converted SNN is based on a pre-trained ANN model. In order to be compatible with the final IF model after conversion, the pre-training of ANN usually adds some constraints, termed as constrained ANN. The constraints include removing bias, allowing only ReLU activation function, changing max pooling to average pooling, and so on <ref type="bibr">Diehl et al. (2015)</ref> and <ref type="bibr">Hunsberger and Eliasmith (2016)</ref>. These constraints usually cause accuracy loss as shown in Fig. <ref type="figure">9(a)</ref>. Then the conversion from the pre-trained ANN to the converted SNN will suffer from further precision loss during the aforementioned signal conversion and the newly incurred parameter adaption such as weight/activation normalization and threshold tuning. On CIFAR10, the accuracy loss for model conversion is significantly larger than that on small MNIST. The influence of simulation time window T is shown in Fig. <ref type="figure">9</ref>(b), wherein the smaller T produces significantly larger accuracy loss. Therefore, the converted SNN usually requires a much longer T to recover the accuracy. In summary, due to the constrained pre-training and lossy model conversion, the converted SNN cannot outperform the natural ANN. More and more recent works benchmark the SNN performance using converted SNNs on ANN-oriented datasets <ref type="bibr">(Diehl et al., 2015;</ref><ref type="bibr">Hu et al., 2018;</ref><ref type="bibr">Sengupta et al., 2019)</ref>. Although this way demonstrates that SNNs can still work well on these ANNoriented workloads, it also makes SNNs lose the opportunity to be the winner.   Table <ref type="table">5</ref> lists the accuracy results of different models on SNNoriented workloads. The natural SNNs (Model-6) can always achieve the best accuracy (98.61% for MLP on N-MNIST, 99.42% for CNN on N-MNIST, and 60.30% for CNN on DVS-CIFAR10) on these networks and datasets, which owes to the natural compatibility with frame-free spike events. In the enforced ANNs, the intensity image-based ANN (Model-5) usually performs better than the binary image-based one (Model-4) except for one occasional case of the CNN on N-MNIST. The reason is similar to that on ANN-oriented workloads, because the relative simplicity of N-MNIST narrows the accuracy gaps between different models. The corresponding training curves of all the models in Table <ref type="table">5</ref> are presented in Fig. <ref type="figure">10</ref>. Overall, on these SNN-oriented workloads, the natural SNNs are able to converge better than other ones converted to process in ANN domain.</p><p>We attempt to explain the accuracy results in a comprehensive way. Fig. <ref type="figure">11</ref> visualizes the different inputs injected to the three models (Model-4/5/6) on SNN-oriented workloads. For the first two models, the spike events are converted to binary images or intensity images according to the signal conversion methods in Fig. <ref type="figure">4</ref>(c) or (d), respectively. Since Model-4 directly receives each spike pattern (0/1) as a binary image, the features are  temporal dimension and then normalizes the spike number to be an intensity value. As Fig. <ref type="figure">11(b</ref>) depicts, the object contour can be kept to a great extent, however, the detailed sharp features are blurred. In contrast to the above two frame-based models, Model-6 naturally receives the spike events. The raw spike pattern at every time step (actually it still has to aggregate the spike events of every 3 ms to narrow the total simulation time length) is visualized in the odd columns of Fig. <ref type="figure">11(c</ref>). Recalling Eq. ( <ref type="formula">4</ref>), the membrane potential in SNNs actually accumulated all the previous inputs with a temporal decay factor if we ignore the potential reset after firing for simplicity. To mimic the equivalent inputs finally injected onto the membrane potential, we plot the decayed accumulation of previous spike patterns at every time step and show them in the even columns of Fig. <ref type="figure">11(c</ref>). It can be seen that the natural SNN (Model-6) is well capable of remaining the detail features contained in the spike events, which is one reason to explain why it can perform better on SNN-oriented workloads.</p><p>Besides the visualization of input signals, we further visualize the activations (in ANNs) and the firing rates (in SNNs) to compare the recognition capability of these three models. Fig. <ref type="figure">12</ref> provides the 2D embedding results generated by a nonlinear dimensionality reduction algorithm, t-distributed stochastic neighbor embedding (t-SNE) <ref type="bibr">(Matten &amp; Hinton, 2008)</ref>. It can project the high-dimensional embedding down to a low-dimensional one for data visualization. Due to the sensitivity of t-SNE to the setting of perplexity (ppl) value, we adopt two ppl values (ppl = 5, ppl = 50) on each dataset according to the suggested range by <ref type="bibr">Matten and Hinton (2008)</ref>. In this way, we project the 256-dimensional activations or firing rates of the last hidden layer down to two dimensions. Points with the same color have the same label and with different color belong to different classes. Therefore, more concentrated points of the same color and more divergent points of different color reflect better classification performance of the model. Although different ppl setting indeed causes pattern change, the relative clustering effects between three models remain consistent as ppl value varies. On N-MNIST, three models are all able to project data from different classes into different point clusters. Compared with Model-4 and Model-5, Model-6 can attract the points within each class more compact and push classes farther, which indicates that Model-6 performs the best. On DVS-CIFAR10, all models present large cluster overlaps. This implies the incapability to achieve comparable accuracy scores like those on the simple N-MNIST task. It is obvious that the clustering effect from Model-4 to Model-6 is gradually strengthened. Combining the results on N-MNIST and DVS-CIFAR10, it confirms the superiority of Model-6 on SNN-oriented workloads, which is consistent with the results in Table <ref type="table">5</ref>. Note that the classification performance of models has been already demonstrated in Tables <ref type="table">4</ref> and<ref type="table">5</ref>, and the t-SNE simulation is just an auxiliary way of visualization for better understanding (not the focus of this work). This is the reason why we only show the t-SNE results for Table <ref type="table">5</ref> with Model 4/5/6 on SNN-oriented datasets and do not repeat the same experiments for Table <ref type="table">4</ref> with Model 1/2/3.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.3.">Cost analysis</head><p>In this subsection, we count the memory and compute costs for all models in Tables <ref type="table">4</ref> and<ref type="table">5</ref>. The cost calculation is according to Eqs. ( <ref type="formula">8</ref>)-( <ref type="formula">9</ref>). The results are shown in Table <ref type="table">6</ref>, wherein the cost values are the average results across all testing samples during inference.</p><p>Memory cost analysis. For all ANN models, the memory cost mainly includes the weight memory and activation memory. Here we omit some possible intermediate variables (e.g. partial sums) for simplicity, since they are related to the hardware architecture design (e.g. dataflow mapping). We also ignore the memory cost for bias because it occupies only a small fraction of the total memory especially in CNNs. On ANN-oriented workloads, the memory cost of SNNs is comparable with that of ANNs. Specifically, the amount of membrane potentials in SNNs is the same with that of activations in ANNs, but the extra spikes of SNNs still consume memory. Fortunately, the spike is in the binary format, i.e. only one bit, the memory cost of which is negligible. On SNNoriented workloads, similar conclusions can be drawn, except for the Model-4 (enforced binary ANN). The input size of Model-4 is much larger than Model-5/6 (see Fig. <ref type="figure">4</ref>), which significantly increases the feature map size and memory/compute cost. Note that the memory cost of ANNs will be increased if we consider the bias memory, especially in the case of MLP networks.</p><p>Compute cost analysis. Different from the memory cost determined by the time step with the maximum memory consumption, the compute cost takes the total operations across all time steps within the entire simulation time window (T ) into account. Two major conclusions are observed: (1) All SNNs can remove the costly multiplication operations (under the mentioned condition of T w = 1) which is very helpful for power and energy reduction;</p><p>(2) The number of addition operations in SNNs might exceed that of ANNs or at least in the same level because SNNs require a processing duration, i.e. T , to accomplish a practical task under the rate coding scheme. The first conclusion does not include Model-3 with encoding layer, whose first layer works in the ANN-SNN hybrid mode, leading to extra multiplications and a bit more additions. The second conclusion is a little counterintuitive against the common sense that the firing activity of SNNs is very sparse, i.e. should be low-cost. It is correct that the spike events are very sparse at each time step, which results in the low-power characteristic of SNN-based devices. However, the compute cost here is determined by the total activities across all time steps, which is more close to the energy consumption rather than power. Note: #-amount, W-weight, A-activation, P-membrane potential, S-spike, M-memory (total), Add-addition, Mul-multiplication. Mul is much more costly than Add in hardware implementation.</p><p>a Refers to the model using probabilistic sampling for input signal conversion.</p><p>b Refers to the model using encoding layer for input signal conversion. On ANN-oriented workloads, the amount of addition operations of SNNs is higher than that of ANNs due to the mentioned time window for rate coding. Specifically, we find Model-3 of enforced SNN consumes less additions than Model-2 of converted SNN. This is because Model-2 requires longer time window (up to 66.7x, see Table <ref type="table">2</ref>) to recover the accuracy loss after model conversion. On SNN-oriented workloads, Model-4 of enforced binary ANN consumes much higher compute cost due to the larger feature size as mentioned earlier. Model-6 of natural SNN removes the costly multiplications; moreover, it consumes lower additions than Model-5 of enforced intensity ANN, which presents an opposed observation compared to the case of Model-1/2/3. This indicates that the spike train of neuromorphic datasets (e.g. N-MNIST and DVS-CIFAR10) is much sparser than that of converted ANN-oriented datasets (e.g. MNIST and CIFAR10). Therefore, the SNN models on neuromorphic datasets can suppress the extra cost caused by the rate coding window.</p><p>From the above analyses, we know that the firing activities in SNNs will greatly affect the accuracy and compute cost. Thus we further conduct sensitivity analysis shown in Fig. <ref type="figure">13</ref>, taking Model-6 over DVS-CIFAR10 dataset as an example. We omit the memory cost since it is only determined by the maximum amount of spike events at a certain time step rather than the total spikes across the entire time window. Two factors, i.e. T and u th , are taken into account. We can get two observations: (1) Apparently, a larger T can usually bring higher accuracy but causes more compute cost; (2) A properly larger u th could improve both the accuracy and compute cost simultaneously. A too small u th will produce much more spikes that not only increases the compute cost but also decreases the model accuracy due to lower pattern discriminability. Although a too large u th could sparsify the spikes significantly, it will compromise the network's expression ability leading to lower accuracy.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.4.">Additional comparison</head><p>Besides SNNs, both RNNs and temporal CNNs in the neural network family are also able to process temporal information. Therefore, here we provide an additional comparison of RNNs, temporal CNNs, and SNNs on N-MNIST and DVS-CIFAR10 datasets, as shown in Tables <ref type="table">7</ref><ref type="table">8</ref>. We implement plain RNN, long short term memory (LSTM), and temporal CNN models, which are in non-spiking version, as well as SNNs in spiking version on these two datasets. We also reference other reported results. All the network structures are detailed to help understanding.</p><p>First, our SNN results are advanced compared with previously reported references, benefit from our selection of superior training algorithm and signal conversion method. The minor accuracy degradation compared to the recent work <ref type="bibr">(Wu et al., 2019</ref>) is because we use a smaller network and do not adopt the neuron normalization and voting-based classification techniques. We would like to emphasize that the focus of this work is to give a comprehensive benchmarking methodology rather than beating prior work, so we abandon the possible optimization techniques. Our LSTM result is consistent with prior work <ref type="bibr">(Liu et al., 2018)</ref>, which also reflects the reliability of our implementation.</p><p>Second, regarding the comparison between SNNs and RNN (LSTM)/temporal CNN models, it seems that the natural SNN InputL-64C3-128C3-AP2 99.42% -128C3-AP2-256FC-10 Note: In temporal CNN, the convolution and pooling here are 3D operations. Here C3 refers to 3 &#215; 3 &#215; 3 convolution, and AP2 refers to 2 &#215; 2 &#215; 2 average pooling. InputL-64C3-AP2-60.3% 128C3-AP2-256FC-10 Note: In temporal CNN, the convolution and pooling here are 3D operations. Here C3 refers to 3 &#215; 3 &#215; 3 convolution, and AP2 refers to 2 &#215; 2 &#215; 2 average pooling. model (Model-6) has greater potential to perform better. Although our natural SNN performs slightly worse than the temporal CNN on N-MNIST, we should note that the latter has much more weight parameters due to the 3D convolution. We try our best to explain the advantages of SNNs on neuromorphic workloads as follows: (1) The temporal integration of historical information can naturally maintain the detail features of sparse data better (see Fig. <ref type="figure">11);</ref><ref type="figure"/> (2) The leakage and thresholded firing mechanism in each neuron can synergistically help denoise redundant information and thus remain useful features <ref type="bibr">(Evangelos et al., 2015;</ref><ref type="bibr">Gutig, 2016)</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.">Conclusion and discussion</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.1.">Brief summary</head><p>In this work, we design well-rounded experiments to answer the questions of ''what workloads are ideal for SNNs and how to evaluate SNNs makes sense''. Specifically, we conduct extensive workload analyses using different benchmark datasets (ANNoriented and SNN-oriented), processing models (ANNs and SNNs), signal conversion methods (images&#8660;spikes), and learning algorithms (direct and indirect supervised training). Comprehensive evaluation metrics are proposed in consideration of the trade-off between the application accuracy and the memory/compute cost.</p><p>Through a variety of comparisons, visualizations, and sensitivity studies, we give the following modeling insights:</p><p>&#8226; On simple ANN-oriented workloads (e.g. MNIST), Model-3 (enforced SNN) is a better choice with acceptable accuracy and less compute cost (without costly multiplications and with slightly more additions).</p><p>&#8226; On more complicated ANN-oriented workloads (e.g. CIFAR10), Model-1 (natural ANN) is preferred to maintain the model accuracy. Although it needs multiplications, the amount of additions is the least one.</p><p>&#8226; On SNN-oriented workloads, Model-6 (natural SNN) is the best selection with both higher accuracy and lower compute cost.</p><p>The converted SNNs on one hand compromise the accuracy compared to their original ANN counterparts due to the extra constraints during ANN pre-training and lossy signal conversion &amp; parameter adaption during SNN conversion; on the other hand also lose efficiency compared to the enforced SNNs due to the much longer time window for accuracy recovery. In SNN modeling, the sparse activities of SNN-oriented datasets significantly benefit the reduction of compute cost, and a properly larger firing threshold can improve both the accuracy and compute cost. In addition, we for the first time point out in the following discussions that the direct porting of ANNs workloads for the evaluation of SNN is unwise although many works are doing so, and a benchmarking framework for SNNs to cover broader tasks, datasets and metrics is urgently needed.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2.">Future opportunities</head><p>Coding schemes and learning algorithms. The application accuracy and execution cost of SNNs are mainly determined by the coding scheme and learning algorithm. Studying new coding schemes beyond the rate coding is an attractive topic to narrow the time window (i.e. towards faster response and lower compute cost) without compromising the expressive power (i.e. maintaining the accuracy). Powerful learning algorithms are also required to further explore the potential of these coding schemes. Moreover, modifying the simple MSE loss function is also a promising way to further improve the network classification performance. All these directions are helpful for training SNNs with less compute cost or extending current models to larger scale, e.g. on ImageNet-level tasks <ref type="bibr">(Deng et al., 2009)</ref>.</p><p>Datasets, benchmarks, metrics, and applications. In this work, we highlight that it is unwise to intuitively test SNNs using ANN-oriented benchmark datasets. More aggressively, it is interesting to ask a question whether current SNN-oriented datasets are neuromorphic enough, i.e. whether they can help fully explore the characteristics of SNNs. In <ref type="bibr">Iyer, Chua, and Li (2018)</ref>, the authors claim that SNNs fail to outperform ANNs even if on neuromorphic datasets (e.g. N-MNIST) and the temporal information in current neuromorphic datasets is not important. We find that their comparison is a little unfair since the network size of the ANNs they use is larger than the SNN baseline. But they still open an interesting question for the neuromorphic community: what is and how to build a good neuromorphic dataset? Despite that there exist different types of generation method for neuromorphic datasets <ref type="bibr">(Li et al., 2017;</ref><ref type="bibr">Mueggler, Rebecq, Gallego, Delbruck, &amp; Scaramuzza, 2017;</ref><ref type="bibr">Orchard et al., 2015;</ref><ref type="bibr">Serrano-Gotarredona &amp; Linares-Barranco, 2015)</ref>, scanning static image datasets into the spiking version using a DVS-like device is still the current common practice. However, it might not be the best way to exploit the spatio-temporal processing capability of SNNs.</p><p>Second, most of the existing benchmarks are based on visual tasks, especially the image recognition. They are insufficient to showcase the capability of SNNs in processing temporal or undeterministic information. We need a broader set of benchmark tasks for SNNs to drive research innovations. Besides visual recognition applications, processing auditory and language information with intrinsic temporal properties or finding fast solution of optimization problems that do not require high solution accuracy, for example, might be well-suited for SNNs. As another example, object detection <ref type="bibr">(Binas, Neil, Liu, &amp; Delbruck, 2017;</ref><ref type="bibr">Zhu, Yuan, Chaney, &amp; Daniilidis, 2018)</ref> in high-volume videos is promising to benefit much from the event-drive paradigm of SNNs towards low cost. Furthermore, more evaluation metrics beyond the proposed recognition accuracy and memory/compute cost are also needed. For instance, how to assess the capability of temporal association, memorization capacity, fault tolerance, and practical running efficiency on devices are all interesting.</p><p>Furthermore, recognizing images is just one out of a huge number of tasks the brain can perform. Extensive efforts to explore more functionality potentials of SNNs should be conducted. One promising direction is to study the network dynamics from a more global angle. Each neuron in SNNs is a dynamic model with firing activity that distinguishes from ANNs, which can bring rich response behaviors in the temporal domain. For example, an SNN could memorize multiple scenarios encoded in metastable states and can output the corresponding response trajectory under different stimuli. Even if for one stimulus, changing the sequence of input spikes can also produce a totally different response. Therefore, how to accurately control the response trajectory via external stimuli is interesting for the exploration of memorization behavior. Overall, SNNs have a huge behavior space that merits in-depth analysis in the future. In this paper we just provide several points to start a discussion. More investigations are expected, requiring efforts from the entire community.</p></div></body>
		</text>
</TEI>
