More Like this

1. Convolutional Neural Network Optimization Using Modified NSGA-II

   Su, Zhidong Su; Sheng, Weihua; Zhang, Senlin (August 2021, The 11th IEEE International Conference on CYBER Technology in Automation, Control, and Intelligent Systems)

   null (Ed.)

   Convolutional Neural Networks (CNN) are becomin deeper and deeper. It is challenging to deploy the networks directly to embedded devices be- cause they may have different computational capacities. When deploying CNNs, the trade-off between the two objectives: accuracy and inference speed, should be considered. NSGA-II (Non-dominated Sorting Genetic Algorithm II) algorithm is a multi-objective optimiza- tion algorithm with good performance. The network architecture has a significant influence on the accuracy and inference time. In this paper, we proposed a con- volutional neural network optimization method using a modified NSGA-II algorithm to optimize the network architecture. The NSGA-II algorithm is employed to generate the Pareto front set for a specific convolutional neural network, which can be utilized as a guideline for the deployment of the network in embedded devices. The modified NSGA-II algorithm can help speed up the training process. The experimental results show that the modified NSGA-II algorithm can achieve similar results as the original NSGA-II algorithm with respect to our specific task and saves 46.20% of the original training time. 

   more » « less
2. Accelerating Neural Network Training using Arbitrary Precision Approximating Matrix Multiplication Algorithms

   https://doi.org/10.1145/3458744.3474050  

   Ballard, Grey; Weissenberger, Jack; Zhang, Luoping (August 2021, 2021 International Workshop on Embedded Multicore Systems)

   Matrix multiplication is one of the bottleneck computations for training the weights within deep neural networks. To speed up the training phase, we propose to use faster algorithms for matrix multiplication known as Arbitrary Precision Approximating (APA) algorithms. APA algorithms perform asymptotically fewer arithmetic operations than the classical algorithm, but they compute an approximate result with an error that can be made arbitrarily small in exact arithmetic. Practical APA algorithms provide significant reduction in computation time and still provide enough accuracy for many applications like neural network training. We demonstrate that APA algorithms can be efficiently implemented and parallelized for multicore CPUs to obtain up to 28% and 21% speedups over the fastest implementation of the classical algorithm using one core and 12 cores, respectively. Furthermore, using these algorithms to train a Multi-Layer Perceptron (MLP) network yields no significant reduction in the training or testing error. Our performance results on a large MLP network show overall sequential and multithreaded performance improvements of up to 25% and 13%, respectively. We also demonstrate up to 15% improvement when training the fully connected layers of the VGG-19 image classification network. 

   more » « less
3. Variational Bayesian Quantization

   Yang, Yibo; Bamler, Robert; Mandt, Stephan (July 2020, Proceedings of Machine Learning Research)

   We propose a novel algorithm for quantizing continuous latent representations in trained models. Our approach applies to deep probabilistic models, such as variational autoencoders (VAEs), and enables both data and model compression. Unlike current end-to-end neural compression methods that cater the model to a fixed quantization scheme, our algorithm separates model design and training from quantization. Consequently, our algorithm enables “plug-and-play” compression with variable rate-distortion trade-off, using a single trained model. Our algorithm can be seen as a novel extension of arithmetic coding to the continuous domain, and uses adaptive quantization accuracy based on estimates of posterior uncertainty. Our experimental results demonstrate the importance of taking into account posterior uncertainties, and show that image compression with the proposed algorithm outperforms JPEG over a wide range of bit rates using only a single standard VAE. Further experiments on Bayesian neural word embeddings demonstrate the versatility of the proposed method. 

   more » « less
4. Variational Bayesian Quantization

   Yang, Y; Bamler, R; Mandt, S. (March 2020, Proceedings of the 37 th International Conference on Machine Learning)

   We propose a novel algorithm for quantizing continuous latent representations in trained models. Our approach applies to deep probabilistic models, such as variational autoencoders (VAEs), and enables both data and model compression. Unlike current end-to-end neural compression methods that cater the model to a fixed quantization scheme, our algorithm separates model design and training from quantization. Consequently, our algorithm enables “plug-and-play” compression at variable rate-distortion trade-off, using a single trained model. Our algorithm can be seen as a novel extension of arithmetic coding to the continuous domain, and uses adaptive quantization accuracy based on estimates of posterior uncertainty. Our experimental results demonstrate the importance of taking into account posterior uncertainties, and show that image compression with the proposed algorithm outperforms JPEG over a wide range of bit rates using only a single standard VAE. Further experiments on Bayesian neural word embeddings demonstrate the versatility of the proposed method. 

   more » « less
5. Disjunctive Threshold Networks for Tabular Data Classification

   https://doi.org/10.1109/OJCS.2023.3282948  

   Wang, Weijia; Qiao, Litao; Lin, Bill (January 2023, IEEE Open Journal of the Computer Society)

   IEEE Open Journal of the Computer Society (Ed.)

   While neural networks have been achieving increasingly significant excitement in solving classification tasks such as natural language processing, their lack of interpretability becomes a great challenge for neural networks to be deployed in certain high-stakes human-centered applications. To address this issue, we propose a new approach for generating interpretable predictions by inferring a simple three-layer neural network with threshold activations, so that it can benefit from effective neural network training algorithms and at the same time, produce human-understandable explanations for the results. In particular, the hidden layer neurons in the proposed model are trained with floating point weights and binary output activations. The output neuron is also trainable as a threshold logic function that implements a disjunctive operation, forming the logical-OR of the first-level threshold logic functions. This neural network can be trained using state-of-the-art training methods to achieve high prediction accuracy. An important feature of the proposed architecture is that only a simple greedy algorithm is required to provide an explanation with the prediction that is human-understandable. In comparison with other explainable decision models, our proposed approach achieves more accurate predictions on a broad set of tabular data classification datasets. 

   more » « less