Artificial Neural Networks (ANN) require a tremendous volume of computations which presents a challenge in implementing deep neural networks (DNNs) on edge devices where resources are at a premium. There has been substantial research in recent years [1], [2] to reduce the latency, power, and number of circuit components to implement efficient and reliable DNNs. One approach is to incorporate stochastic computing (SC) for inference on edge devices [3], [4]. In SC-based ANN, inputs, activations and tunable parameters such as weights are represented by a bitstream of "1"s and "0"s having a specific probability of observing "1". Matrix vector multiplications, the large computational loads of ANN, are replaced by probabilistic mathematics in SC, thus greatly reducing the hardware resources otherwise utilized in conventional computing based approaches [5], [6]. Several implementations of neural networks employing SC-multilayer perceptron (MLP) [7], [8], radial basis function NNs [9], convolutional neural networks (CNN) [10], deep belief networks (DBN) [11], and recurrent neural networks (RNN) [12] report competitive accuracy with significantly lower hardware cost and energy consumption [13]. Furthermore, SC remains an attractive choice for specific image processing tasks, such as, local image thresholding [14], low-precision real-time image processing [15] or kernel density approximation [16] due to faster and lower power implementation [17].
Efficient application of SC-ANN on edge devices relies on generating random numbers in an energy-efficient manner. Semiconductor (CMOS) circuits such as 32-bit linear feedback shift register (LFSR) can generate pseudo-random number which requires more than 1000 transistors making it resource-intensive [18]. In addition, non-volatile memory (NV) based technologies including phase change memory (PCM) [19], resistive random access memory (RRAM) [20] are being explored, along with the use of magneto resistive random access memory (MRAM) bits such as magnetic tunnel junctions (MTJs) [17]. MTJ is an attractive choice as it can generate stochastic bitstreams with compact design and low power [21]. Furthermore, MTJs are integrated with CMOS circuits in state-of-the-art semiconductor manufacturing foundries, making their states accessible with higher speed. The generated bitstreams can be tuned using techniques such as applying spin transfer torque (STT) [22] or spin-orbit torque (SOT) [23]. SC-based ANN using MTJ-generated bitstreams is reported in [24] for hand-written digit classification tasks. The stochastic bitstreams of continuous probabilities are generated using an analog approach [24]. However, to achieve ANN parameters (i.e. weights and biases) with analog precision (i.e. 32-bit) using non-volatile and other technologies often requires a lot of circuit overhead [25]- [27] and is more prone to device-to-device variations which can negatively impact the accuracy and scalability of the ANN [25]- [27]. Interestingly, ANN with extremely low precision or quantized weights and biases (can be even binary) are shown to achieve competitive accuracies [28]- [31] compared to 32-bit precision ANN. The accuracy degradation due to the quantization loss in such ANN is prevented during the learning stage by preserving the weight gradients [28]. The idea of quantization can be extended to SC where bitstreams with only a few probabilities are sufficient (i.e. 5 or 11-states) to retain the ANN accuracy. This way the peripheral circuit overhead and the complexity of the stochastic bitstream generating devices can be significantly reduced. Quantization-aware training using hardware-accelerators such as non-volatile computational devices arranged in crossbar architecture are reported with competitive accuracies [29]- [33], where matrix vector multiplications are performed by Kirchhoff's and Ohm's laws. In this paper, we explore the quantization aware training of SC-ANN where the inputs, activations and weights are represented by bitstreams of only a few quantized probabilities. The bitstream values with different probabilities are obtained using experiments where we bias the MTJs with different STT current densities [24]. During the quantization aware training, multiplication operations are simulated using bitwise XNOR operation (bipolar weights) and the addition operation with parallel counters. Furthermore, the sigmoid activation functions are implemented by using a much simpler look up table (LUT) based design due to the fact that the neuron output in quantized training can only assume a few discrete values (5 or 11 states). We tested the performance of our proposed training approach of the SC-based ANN on handwritten digit classification tasks using MNIST datasets where we evaluate the inference performance of such quantized SC-ANN on different topology ANNs (with 1 and 3 hidden layers) and explored the impact of smaller length bitstreams (such as 100-bits) on the accuracy. Using our approach with smaller size bitstreams (i.e. 100 bits length), we show that we can achieve competitive accuracy on MNIST handwritten digit classification tasks compared to 32-bit precision similar architecture ANN with much less energy consumption and latency. In addition, our approach requires low precision (i.e. 3-bit) digital to analog converter (DAC) which can potentially reduce prohibitive energy cost.
The rest of the paper is organized as follows. Section II provides a brief overview of stochastic computing implemented with random bit streams generated by s-MTJs, section III discusses the structure of deep neural networks and quantization, section IV presents a performance analysis and section V presents the conclusion.
In stochastic computing (SC), a number is represented by the number of "1"s in its bitstream. Specifically, in unipolar encoding, the value of a number corresponds to the probability "1"s in its bitstream [34]. If a number X has N number of 1s and bitstream length is L, then stochastic representation of X will be P(X)= N/L. Multiplication of two numbers, X=A×B can be performed using simple AND gates, as in stochastic domain since P(X)=P(A). P(B). Thus, the number of transistors is significantly reduced. Unipolar encoding can only represent positive numbers in the range [0, 1]. However, bipolar encoding is required to represent both positive and negative numbers in the range [-1, 1] [34]. In this scheme, a number is represented by X= 2× P(X) -1 or alternatively, P(X)= (X + 1)/ 2.
To accommodate negative weights in the neural network, bipolar encoding has been used. XNOR gate can be used for multiplication of two bipolar numbers since P(X)= P(A).P(B) + P(A) . P(B) and for bipolar encoding it becomes
which produces X=A.B [34]. Thus, the number of transistor gates required for arithmetic operations is also reduced significantly in this case.
Using CMOS devices as shown in Figure 1(a), a random number generator can convert binary numbers to stochastic bitstreams. An alternative and more energy efficient way of generating Random Numbers is by using a MTJ shown in figure 2, consisting of two ferromagnetic layers separated by an oxide layer [24]. One layer has fixed magnetic orientation, referred to as the reference layer. The other layer's orientation can be switched. If the reference layer and free layer have magnetic orientation in the same direction, it corresponds to low resistance referred to as the Parallel (P) state ('0'). When the free layer has magnetic orientation in the opposite direction, it corresponds to higher resistance and is referred to as the Antiparallel (AP) state ('1') [35]. The two states are separated by an energy barrier E b and corresponding retention time 𝜏𝜏 = 𝜏𝜏 0 exp � 𝐸𝐸 𝑏𝑏 𝑘𝑘 𝐵𝐵 𝑇𝑇 � �. Here 𝑘𝑘 𝐵𝐵 denotes the Boltzmann constant and T denotes temperature. The energy barrier is determined by the interfacial magnetic anisotropy in the case of perpendicular magnetic tunnel junctions [36]- [38]. If 𝐸𝐸 𝑏𝑏 is chosen to be small enough, then random thermally activated switching may occur at room temperature. For example, 𝐸𝐸 𝑏𝑏 < 16𝑘𝑘 𝐵𝐵 𝑇𝑇 resulted in bitstream generation at a rate of milliseconds in [24], although much faster bit generation has been demonstrated in devices with smaller energy barriers or devices where the energy barrier is temporarily reduced by voltage controlled magnetic anisotropy (VCMA) [24], [39]- [41]. [42] In prior work, different bias voltages were applied to generate bitstreams corresponding to different input and weight values. It was shown that tunability from > 95% AP to > 95% P was achieved by bias voltages from -1V to +1V (figure 2c) [24]. About 30 different bias voltages were used to generate 30 different bitstreams per MTJ. The products (XNOR) of two MTJs were then utilized to increase resolution of each bitstream [24] and used for inference (not for learning with backpropagation) of a neural network for recognition of handwritten digits from MNIST dataset. However, this work [24] did not consider quantization of synaptic weights and outputs, which is implemented in our current work with only 5 or 11 quantized states. Such quantization could lead to significant improvements in latency and energy efficiency for inference, without significantly sacrificing inference accuracy.
Two neural networks were implemented so the performance between the quantized and high precision implementation could be compared for a single hidden layer network and deeper network with three hidden layers. The input layer for both consists of 28×28 =784 pixels or data points. Architectures are summarized in Table I below. The network with one hidden layer is visualized in Figure 1. For the inference in the DNN, the were implemented using XNOR gates. For addition in binary domain, approximate parallel counter [43] or scaled multiplexer [44] can be used since SC-based networks generally allow some degree of precision loss. The activation function can be implemented for stochastic bitstream in various ways such as sigmoid and tanh functions implemented using JK Flip Flop or Finite State Machine (FSM) [45]. However, due to shorter bitstream and highly quantized (only 5 and 11 states) networks, parallel counter-based addition and look up table based sigmoid can be adopted without significant resource utilization, as explained in the following section.
Quantization: For practical implementation on edge devices, it would be resource intensive to generate all the probabilities. Hence, quantized weights have been used. In this network, weights have been scaled in the range of [-1, +1]. Weights or nodes are further quantized into 5 states or 11 states. We chose odd number of quantization states as they are centered around zero. Also, our past work [29], [31] showed that there is a significant gain in accuracy as one goes from 3-states to 5-states but gain is less significant beyond 5 states. Thus, the choice of 5-states was with an aim of achieving reasonable accuracy and the choice of 11-states was to see if there is a significant improvement in accuracy beyond 5-states. These quantized states are converted to corresponding stochastic bitstream for multiplication operation with bitwise XNOR. Afterwards, addition was performed with parallel counters. For implementation of the activation function, it is noticed that sigmoid activation function has only 3 or 6 distinct states (figure 4a) corresponding to 5 or 11 quantized states systems respectively. Hence sigmoid activation is implemented with look up tables.
After completion of feedforward pass, error is calculated from difference of predicted and actual output.
Here we have followed the approach of [29], [46], where n is the number of quantized states and a and b represent lower and upper limits.
Weights and nodes of the neural network are converted to corresponding stochastic bitstreams for multiplication operation with bitwise XNOR, after which addition is performed with a parallel counter. Sigmoid activation has only several distinct states so it can be implemented with look up tables (figure 4b).
Learning: For quantization-aware learning, methods of [29] were incorporated. Equations for learning are as follows.
Cost function, C =
Gradient of cost function for output layer or error,
Error for preceding layer,
Weight update parameter, ∆W ij = ηx i l δ j l+1 f l+1 ′ (7) Here η= learning rate, f l+1 ′ = gradient of the activation function of layer l+1 neuron.
Here cost function is calculated by summation of square of error function defined by differences between predicted output and actual output (Eq. 4). Gradient of cost function for output layer is calculated by differentiation with respect to predicted outputs which can be termed as error (Eq 5). For preceding layers, errors were calculated by the backpropagation Eq. 6. Weights are updated as per Eq 7. Gradient of activation function is not included for back propagation though it is incorporated for weight updates [29]. It is to be noted that since differentiation for discrete or quantized values produce zero gradients, straight through estimator approach has been applied to backpropagate gradients [28], [29]. Algorithm: Quantization aware training for SC-ANN Input:
// X, T are n dimensional vectors. // number of layers= L, learning rate= η, QL= number of quantization states.
Weight initialization: W ← Gaussian distribution [-1, 1] X ← X/ maximum (X); // input data normalized;
for i= 1 to num_neurons do
j= number of inputs end for // stochastic multiplication implemented with XNOR for bipolar stochastic bitstreams // sigmoid activation implemented with look up tables. end for // Compute gradient Compute gradient G L = 𝜕𝜕𝜕𝜕 𝜕𝜕𝑋𝑋 𝐿𝐿 from X L and X 0
η ← λη // λ denotes learning rate decay end for
As mentioned earlier, neural networks with one hidden layer and three hidden layers were tested with the MNIST dataset. Each network was tested for 5 quantized states and 11 quantized states for weights and hidden layer nodes. For stochastic Computing (SC), bitstream lengths of 100, 200, 300, 400 and 500 were used. Accuracy was determined for two scenarios for different bitstreams: Accuracy: i. Method 1: Training was conducted for 5 and 11 quantized states across both networks without using SC. Inference computations were carried out both without SC and with SC, utilizing the mentioned bitstreams.
ii. Method 2: To explore the improvement of inference accuracy via SC training, the forward pass was carried out with SC and quantization while the backpropagation was carried out with high accuracy wights (not SC). Afterwards inference was conducted with and without SC, like the previous method.
The accuracy of the networks for different training and inference combinations are shown in figure 6.
For one hidden layer architecture, accuracy is always higher for training with SC. Besides, longer bitstreams generally provide higher accuracy. For 5 quantized states, the highest accuracy of 93.68% was achieved for 400 bits while for 11 states, highest accuracy of 94.48% was obtained for 500 bits.
For three hidden layer architecture, higher accuracy was also achieved for training with SC for all cases, similar to the one hidden layer architecture. However, the relationship between bitstream length and accuracy is not that straightforward for the deeper network. The highest accuracy obtained for 5 states is 96.62% for 300 bits and for 11 states it is 96.82% for 400 bits. It can be theorized that noise arising from quantization and stochasticity errors assists the model in escaping the local minimum and moving closer to the global minimum of the loss function for the one hidden layer architecture. However, in a three-hidden-layer architecture, the cumulative effect of noise may become too large, potentially averaging itself out and failing to contribute effectively to approach the global minimum.
Besides, in the three hiddn layer architecture, the accuracy is constrained by number of states. Hence it is unlikely that longer bitstream can attain any considerable improvement beyond 96.82% (highest attained accuracy for the systems). We also note that for a system where inference acuracy of 92% is sufficient, 100 bit SC system with one hidden layer architecture may be adequate while for a system with required accuracy of 96%, 100 bit SC system with three hidden layer architecture can be used.
The performance of these networks can be compared with other contemporary works. Reference [24] achieved 95% accuracy for 1024 bit long stream. In this work, 94.33% accuracy was obtained for 500 bits for 11 quantized weights for one hidden layer architecture. Daniels et al reported 97% accuracy using Lenet-5 architecture with bitstream length of 128 [25]. They used traditional digital circuitry to conrol bitstream statistics of SMTJ. Lenet-5 incorporates 6 hidden layers with multiple convolutional layers. Unipolar encoding for SC was incorporated, hence separate excitatory and inhibitory subnetworks were used to accommodate positive and negative weights. They used resolution of One partiular distinction is that while we have incorporated quantization and SC for both training (forward pass) and inference, [24] and [25] used quantization and SC only for inference. SC inherently offers resilience to random bit flips to some extent, as a single random bit flip in the SC domain has less impact on the system compared to the binary domain, especially when the bit flip affects the most significant bit [47].
Energy consumption for conventional CMOS based LFSR RNG per bit is ~10fJ [25]. For MTJ devices, energy dissipation is related to the retention time and the energy barrier E b between parallel and anti-parallel states. For a retention time of τ ≈ 10 ns, the energy per bit is ∼20 fJ assuming an applied voltage of ∼1 V and device resistance of 500 kΩ, which is comparable to CMOS-only RNGs [24]. With τ < 1 ns, the energy can be reduced further [24].
We experimented with different bitstreams lengths. In [24] the authors reported 95% accuracy for 1024 bit long stream. We achieved 96.44% with 100 bits long stream with three hidden layers or 94.48% with 500 bits for one hidden layer. Thus, bitstream generation energy can be lowered from ~2100 nJ to 810~820 nJ for comparable accuracy with 2.6× reduction in energy. More importantly, our approach eliminates the requirement of digital to analog converter (DAC) which contributes to probably the biggest energy saving.
MTJ controlling circuitry also consumes energy. To generate bitstreams with deeper resolution in their respective probabilities, a complex combination of MTJ bitstreams and additional XNOR operations are required. In our proposed approach, bitstreams with only 5 or 11 different probabilities will suffice, which greatly simplifies the control circuitry and can be generated by biasing the MTJ with 5 or 11 different voltages.
Our quantized approach also comes with additional benefit in implementing the neuron activations such as sigmoid used in our works. As the neurons and weights assume only a smaller number of discretized states, the sigmoid can be implemented using a simple look-up-table (LUT), which will output only a few states. To test the latency improvement, we tested sigmoid activation implemented on FPGA using Xilinx software tools. A total of 9 clock cycle was needed to compute sigmoid using 11-state LUT while 80 clock cycle was required for similar 32-bit precision computation.
SC-ANN with s-MTJ sourced stochastic bitstream has been tested for MNIST dataset with different bitstream lengths and quantized states. It was observed that even with shorter bitstreams of 100 bits and only 5 or 11 quantized states accuracy of 92% to over 96% was obtained. This approach achieved 9× improvement in energy and 2.6× improvement in latency, making it highly advantageous for neural network inference in edge computing applications.