skip to main content


Title: Low-Energy Deep Belief Networks using Intrinsic Sigmoidal Spintronic-based Probabilistic Neurons
A low-energy hardware implementation of deep belief network (DBN) architecture is developed using near-zero energy barrier probabilistic spin logic devices (p-bits), which are modeled to real- ize an intrinsic sigmoidal activation function. A CMOS/spin based weighted array structure is designed to implement a restricted Boltzmann machine (RBM). Device-level simulations based on precise physics relations are used to validate the sigmoidal relation between the output probability of a p-bit and its input currents. Characteristics of the resistive networks and p-bits are modeled in SPICE to perform a circuit-level simulation investigating the performance, area, and power consumption tradeoffs of the weighted array. In the application-level simulation, a DBN is implemented in MATLAB for digit recognition using the extracted device and circuit behavioral models. The MNIST data set is used to assess the accuracy of the DBN using 5,000 training images for five distinct network topologies. The results indicate that a baseline error rate of 36.8% for a 784x10 DBN trained by 100 samples can be reduced to only 3.7% using a 784x800x800x10 DBN trained by 5,000 input samples. Finally, Power dissipation and accuracy tradeoffs for probabilistic computing mechanisms using resistive devices are identified.  more » « less
Award ID(s):
1739635
NSF-PAR ID:
10057857
Author(s) / Creator(s):
; ; ; ;
Date Published:
Journal Name:
ACM Great Lakes Symposium on VLSI
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Magnetic Random-Access Memory (MRAM) based p-bit neuromorphic computing devices are garnering increasing interest as a means to compactly and efficiently realize machine learning operations in Restricted Boltzmann Machines (RBMs). When embedded within an RBM resistive crossbar array, the p-bit based neuron realizes a tunable sigmoidal activation function. Since the stochasticity of activation is dependent on the energy barrier of the MRAM device, it is essential to assess the impact of process variation on the voltage-dependent behavior of the sigmoid function. Other influential performance factors arise from varying energy barriers on power consumption requiring a simulation environment to facilitate the multi-objective optimization of device and network parameters. Herein, transportable Python scripts are developed to analyze the output variation under changes in device dimensions on the accuracy of machine learning applications. Evaluation with RBM circuits using the MNIST dataset reveal impacts and limits for processing variation of device fabrication in terms of the resulting energy vs. accuracy tradeoffs, and the resulting simulation framework is available via a Creative Commons license. 
    more » « less
  2. This paper presents a design approach for the modeling and simulation of ultra-low power (ULP) analog computing machine learning (ML) circuits for seizure detection using EEG signals in wearable health monitoring applications. In this paper, we describe a new analog system modeling and simulation technique to associate power consumption, noise, linearity, and other critical performance parameters of analog circuits with the classification accuracy of a given ML network, which allows to realize a power and performance optimized analog ML hardware implementation based on diverse application-specific needs. We carried out circuit simulations to obtain non-idealities, which are then mathematically modeled for an accurate mapping. We have modeled noise, non-linearity, resolution, and process variations such that the model can accurately obtain the classification accuracy of the analog computing based seizure detection system. Noise has been modeled as an input-referred white noise that can be directly added at the input. Device process and temperature variations were modeled as random fluctuations in circuit parameters such as gain and cut-off frequency. Nonlinearity was mathematically modeled as a power series. The combined system level model was then simulated for classification accuracy assessments. The design approach helps to optimize power and area during the development of tailored analog circuits for ML networks with the ability to potentially trade power and performance goals while still ensuring the required classification accuracy. The simulation technique also enables to determine target specifications for each circuit block in the analog computing hardware. This is achieved by developing the ML hardware model, and investigating the effect of circuit nonidealities on classification accuracy. Simulation of an analog computing EEG seizure detection block shows a classification accuracy of 91%. The proposed modeling approach will significantly reduce design time and complexity of large analog computing systems. Two feature extraction approaches are also compared for an analog computing architecture. 
    more » « less
  3. In this paper, a probabilistic interpolation recoder (PIR) circuit is developed for deep belief networks (DBNs) with probabilistic spin logic (p-bit)-based neurons. To verify the functionality and evaluate the performance of the PIRs, we have implemented a 784 × 200 × 10 DBN circuit in SPICE for a pattern recognition application using the MNIST dataset. The PIR circuits are leveraged in the last hidden layer to interpolate the probabilistic output of the neurons, which are representing different output classes, through sampling the p-bit’s output values and then counting them in a defined sampling time window. The PIR circuit is proposed as an alternative for conventional interpolation methods which were based on using a resistor capacitor tank to integrate each neuron’s output, followed by an analog-to-digital converter to generate the digital output. The circuit simulation results of PIR circuit exhibit at least 54%, 81%, and 78% reductions in power, energy, and energy-error-product, respectively, compared to previous techniques, without using any of the area-consuming analog components in the interpolation circuit. In addition, PIR circuits provide an inherent single stuck at fault tolerant feature to mitigate both transient and permanent faults at the circuit’s output. Reliability properties of the PIR circuits for single stuck-at faults are shown to be enhanced relative to conventional interpolation without requiring hardware redundancy. 
    more » « less
  4. Abstract

    Employing the probabilistic nature of unstable nano-magnet switching has recently emerged as a path towards unconventional computational systems such as neuromorphic or Bayesian networks. In this letter, we demonstrate proof-of-concept stochastic binary operation using hard axis initialization of nano-magnets and control of their output state probability (activation function) by means of input currents. Our method provides a natural path towards addition of weighted inputs from various sources, mimicking the integration function of neurons. In our experiment, spin orbit torque (SOT) is employed to “drive” nano-magnets with perpendicular magnetic anisotropy (PMA) -to their metastable state, i.e. in-plane hard axis. Next, the probability of relaxing into one magnetization state (+mi) or the other (−mi) is controlled using an Oersted field generated by an electrically isolated current loop, which acts as a “charge” input to the device. The final state of the magnet is read out by the anomalous Hall effect (AHE), demonstrating that the magnetization can be probabilistically manipulated and output through charge currents, closing the loop from charge-to-spin and spin-to-charge conversion. Based on these building blocks, a two-node directed network is successfully demonstrated where the status of the second node is determined by the probabilistic output of the previous node and a weighted connection between them. We have also studied the effects of various magnetic properties, such as magnet size and anisotropic field on the stochastic operation of individual devices through Monte Carlo simulations of Landau Lifshitz Gilbert (LLG) equation. The three-terminal stochastic devices demonstrated here are a critical step towards building energy efficient spin based neural networks and show the potential for a new application space.

     
    more » « less
  5. The ever increasing size of deep neural network (DNN) models once implied that they were only limited to cloud data centers for runtime inference. Nonetheless, the recent plethora of DNN model compression techniques have successfully overcome this limit, turning into a reality that DNN-based inference can be run on numerous resource-constrained edge devices including mobile phones, drones, robots, medical devices, wearables, Internet of Things devices, among many others. Naturally, edge devices are highly heterogeneous in terms of hardware specification and usage scenarios. On the other hand, compressed DNN models are so diverse that they exhibit different tradeoffs in a multi-dimension space, and not a single model can achieve optimality in terms of all important metrics such as accuracy, latency and energy consumption. Consequently, how to automatically select a compressed DNN model for an edge device to run inference with optimal quality of experience (QoE) arises as a new challenge. The state-of-the-art approaches either choose a common model for all/most devices, which is optimal for a small fraction of edge devices at best, or apply device-specific DNN model compression, which is not scalable. In this paper, by leveraging the predictive power of machine learning and keeping end users in the loop, we envision an automated device-level DNN model selection engine for QoE-optimal edge inference. To concretize our vision, we formulate the DNN model selection problem into a contextual multi-armed bandit framework, where features of edge devices and DNN models are contexts and pre-trained DNN models are arms selected online based on the history of actions and users' QoE feedback. We develop an efficient online learning algorithm to balance exploration and exploitation. Our preliminary simulation results validate our algorithm and highlight the potential of machine learning for automating DNN model selection to achieve QoE-optimal edge inference. 
    more » « less