Deep neural networks (DNNs) and machine learning techniques have achieved great success in solving several traditionally challenging problems [1]- [3] such as computer vision [4], [5] and video recognition [6], [7]. DNNs are widely used today in numerous applications, from recommender systems [8] to autonomous vehicles [9], [10]. However, the execution platform of several of these applications lacks the resources for the efficient execution of DNNs, such as inexpensive robots [11]- [14], unmanned aerial vehicles (UAVs) [15], [16], and Internetof-things (IoT) devices [17]. This is because the fast prediction of DNNs (i.e., inferencing) is a resource-intensive task [18] that requires energy, large memories, and capable processors. The traditional solution to this problem is to offload all the computations to the cloud. Nevertheless, such offloading is not possible in several situations because of privacy concerns [19]- [22], limited Internet connectivity, or tight-timing constraints (e.g., home video recordings, drones, and robots surveying a disaster area).
To amortize the cost of DNN, researchers have studied several machine learning techniques, such as weight pruning [18], [23]- [25], quantization [26]- [29], and mixed precision inferencing [30]. Furthermore, in-the-edge inferencing enforces the computation of single-batch inference because of the limited number of available requests in a given time.
Compared to multi-batch inferencing (the common practice in cloud servers), single-batch inferencing increases memory footprint, which is an added challenge for in-the-edge devices. Therefore, edge specific DNN frameworks, such as Tensor-Flow Lite [31] and TensorRT [32], integrate and adapt aforementioned techniques for single-batch inference to achieve high performance. Moreover, companies have started to build device-specific frameworks for efficient DNN execution, such as Microsoft ELL [33] for Raspberry Pis [34] with several compiler-and software-level optimizations. However, using only software techniques cannot guarantee the fast execution of DNNs. This is because current hardware platforms are not specifically designed for DNNs, the execution of which has unique characteristics. This inefficiency of general-purpose hardware platforms for DNN execution has led to specialized hardware designs and ASIC chips targeting high-performance computing (HPC). Additionally, companies have also released specialized accelerator devices for performing fast in-the-edge inferencing, such as Google's EdgeTPU [35], Nvidia's Jetson Nano [36], and Intel's Movidius Neural Computer Stick [37].
This paper presents a unified comparison between commercial edge devices (Table III) with the same set of assumptions among several frameworks (Table II) with the same DNN models (Table I). While focusing on edge-specific singlebatch inferencing, we analyze and compare the performance of several widely used frameworks and their supported optimizations for edge devices. To gain better insights, we profile the software stack of two widely used frameworks (TensrorFlow and PyTorch) on two CPU-and GPU-based edge devices (Raspberry Pi 3B and Jetson TX2). Additionally, we study accelerator-oriented frameworks for edge devices, such as Movidius toolkit. Besides the characterization of several edge devices, to the best of our knowledge, this is the first characterization of EdgeTPU and Jetson Nanofoot_0 . In addition to the performance characterization of edge devices, we investigate whether HPC-level devices (Xeon and HPC GPUs) are a good candidate for single-batch inferencing. Finally, by using a power analyzer and thermal camera, we measure the energy per inference and temperature behavior of edge devices. Our experiments are reproducible and extendable to new platforms by utilizing our open source project on GitHubfoot_1 . The following are the key contributions of this paper:
(2) TensorFlow-Lite: TensorFlow-Lite [31] (TFLite) is the wrapper of the TensorFlow engine for mobile and IoT devices. To reduce the memory and computation footprints, TFLite performs various optimizations on the computation graph and weights of the model. For instance, TFLite offers pruning, structured pruning, and quantization (both post-training and during the training). For deployment, TFLite freezes the computation graph by removing several redundant and unnecessary operations (e.g., converting variable to constant operations).
(3) Keras: Keras [57] is a high-level deep learning API that is built on top of TensorFlow. It is written in Python and is released under the MIT license. The interface of Keras has been developed to provide easy and fast prototyping and to minimize the idea-to-result time. Keras is now integrated into TensorFlow. Therefore, for some models for which Keras has an implementation, we use Keras and TensorFlow implementations interchangeably. (4) Caffe/2: Caffe2 [58] is the successor and a lightweight version of Caffe [59], an academic endeavor, now supported by Facebook. While Caffe supports deploying CNN models on clusters or mobile devices, it is more useful for large-scale use cases rather than mobile, distributed computation, and reduced precision computation use cases. To this end, Caffe2 supports large-scale distributed training, mobile deployment, new hardware, and quantized computation. Moreover, Caffe2 is intended to be modular to enable the fast prototyping of ideas and scalability. Caffe and Caffe2 are written in C++ and CUDA and are open sourced under the Apache 2.0 license. Caffe2 offers Python APIs for its engine.
(5) Movidius NCSDK Toolkit: The neural computing device of Intel, known as Movidius Neural Compute Stick (see Section IV), requires its compatible toolkit, Movidius Neural Compute SDK (NCSDK) [52], to run DNNs. Since the optimizations supported by the NCSDK toolkit are handtuned, importing and compiling a new model is a strenuous process [60] (NCS2 [61], announced after this paper acceptance, claims supporting popular frameworks). (6) PyTorch: PyTorch [62], the Python version of Torch, a computational framework written in Lua, was open-sourced by Facebook in 2017 under the BSD license. Caffe2 (in C/C++) was merged into PyTorch in Facebook in 2018. Since Torch was originally developed as an academic project, it features a large number of community-driven packages and tools. PyTorch, in contrast with TensorFlow, closely represents the Python scientific computing library (i.e., numpy). Internally, PyTorch constructs dynamic computation graphs, which means that during each inference, the computation graph is defined, utilized, and freed. Thus, constructing the entire graph is not necessary for execution. The advantage of such dynamic graphs is the efficient use of memory in cases where the input may vary. On the other hand, the disadvantage of dynamic graphs is the limited opportunities for global optimizations because the entire graph is not known during the execution. (7) TensorRT: NVidia TensorRT [32] is built on CUDA and includes several optimizations to deliver high throughputs and low latencies for DNN applications. The focus of TenorRT Half-Precision † Main interfacing language. *,**,*** More stars represents higher/easier/better. ‡ Support for mixed-precision inferencing, and not training. § TensorFlow Eager execution [55] provides dynamic graphs, but for debugging purposes. After the acceptance of this paper, a beta version of TensorFlow 2.0 [54] has been announced that supports dynamic graphs. ‡ ‡ Every framework supports pruning by zeroing out weights. Here, we show if a framework can automatically benefit from such fragmented weights. † † Experimental implementation. is end-user performance. Therefore, TensorRT can optimize models that are trained on other frameworks after they are imported. Besides mixed-precision computation and quantization in INT8 and FP16, TensorRT tries to minimize the memory footprint by reusing memory and fusion operations. Although all other major frameworks support some part of these optimizations, we find TensorRT to be more user-friendly and easier to deploy. (8) DarkNet: DarkNet [63] is a standalone open-source framework written in C and CUDA. The project code base is small and thus is understandable and modifiable. Since DarkNet is written in C without using high-level languages, it is a good candidate for low-level hardware and software optimizations and creating micro-benchmarks. (9) FPGA Frameworks: To utilize an FPGA-based edge device, PYNQ board [64], we use a set of frameworks, FINN [65] and TVM VTA [66], [67] stack. FINN uses binarized weights for its implementations. TVM uses a customized RISC-based instruction set but on tensor registers. VTA deploys a custom-hardware design on the FPGA by utilizing a PYNQ overlay [68], which enables the offloading of predefined functions to the FPGA. Then, by utilizing the TVM just-in-time compiler, a user can offload the computations of a DNN model to the FPGA without interacting with any hardware-level code. Jetson TX2 [69] Jetson Nano [36] EdgeTPU [35] Movidius NCS [37] PYNQ : Raspberry Pi 4B [70], with 4-core Ctx.A72 and maximum of 4 GB LPDDR4, was released after this paper acceptance. With better memory technology and out-of-order execution, Raspberry Pi 4B is expected to perform better.
Intel Neural Compute Stick 2 [61] with a new VPU chip and support for several frameworks was announced during paper submission, but the product was not released.
This section compares the preceding frameworks from various points of view. In addition to some general aspects such as their programming language, development support from industry, training capability, usability, supported computations for models, and available documentations, Table II summarizes some relative technical advantages and disadvantages of the aforementioned frameworks 3 . For instance, TFLite and Movidius (i.e., NCSDK) require extra steps to configure a model for execution. As a result of such steps, both frameworks provide more optimizations for edge devices. For example, quantization-aware training and freezing a graph in TFLite considerably reduce the memory footprint of the DNN models implemented by TensorFlow. TensorRT also provides the same set of optimizations with less programming effort (and thus less aggressive benefits) with auto-tuning. On the other hand, the full support for mobile-device deployment is available only on TFLite, and partially on other frameworks such as Caffe2.
Low-level modifications, such as changing data ordering, optimizing for cache hits, and performing kernel fusion, are cumbersome tasks. For all DNN frameworks, such modifications are even harder because of their implementation in a high-level language (except DarkNet, which is written in C). Another aspect is the compatibility of frameworks with each other (e.g., the same model description format as input and similar output format). For this paper, we work with several frameworks. Unfortunately, we find limited compatibility among frameworks. Additionally, each framework assumes a set of hyperparameters that might be different in other frameworks. TensorRT provides better compatibly in importing models from other frameworks (including ONNX format); however, this is mainly because TensorRT is an inference-oriented framework, the main purpose of which is efficient inferencing. Similarly, TFLite is an inference-oriented framework backed by TensorFlow training procedures and its model description.
We also study the implemented state-of-the-art optimizations in each framework in their code base. Particularly, we study weight quantization, mixed-precision inferencing, dynamic construction/deconstruction/allocation of computation graph, ability to leverage pruned wights, kernel fusion for efficient caching and use of computing units, auto-tuning to hardware platforms, and support for half-precision numbers (FP16). Each optimization can be implemented in various methods with numerous tradeoffs. Additionally, several of such optimizations can be implemented by users in their application. In Table II, we address only the existence of such optimizations that are officially implemented by the authors of the each framework. Quantization to common datatypes is implemented for all frameworks that are supported by the industry. Quantization is popular because, as several studies show [18], [27], [28], using smaller datatypes can significantly reduce pressure on memory and storage in all systems. Furthermore, implementing quantization to a fixed data type is a fairly easy task for industry. On the other hand, mixed-precision inferencing requires several software-hardware co-designs, and the benefit is limited to specific hardware platforms (e.g., Nvidia Turing µArchitecture [71]) that can exploit such an imbalance in their computational units.
The dynamic manipulation of computation graphs enables efficient memory reuse and minimizes memory footprint. Therefore, increasing the performance of inference. TensorRT supports dynamic graphs, and PyTorch supports them by design choice. The static computational graph could also be beneficial, for example, by enabling offline optimizations (e.g.,TFLite). Thus, both options are viable choices for constructing a computational graph. Another optimization, weight pruning [18], [23]- [25], [72], [73], reduces the storage and computation footprints by removing some of the network connections. However, a framework needs to take further steps to exploit these benefits (e.g., change data representation format to match underlying hardware [73]). This is because data and computations are now in sparse format. Although all frameworks benefit from pruning in reducing their storage footprint, only TensorFlow, TFLite, and TensorRT take further steps in exploiting it for sparse computations. Another essential optimization is kernel fusions [74], which reduces memory traffic by reusing data that are already populated and eliminating redundant loads and stores. The effectiveness of kernel fusions depends on several factors, such as hardware resources and applied software techniques. Therefore, kernel fusions are mostly supported on platformspecific frameworks (e.g., TFLite, Movidius SDK, and Ten-sorRT). Usually, each framework requires careful tuning to particular hardware by an experienced engineer. After tuning, the models and weights are released for use by users. However, in this approach, generally, users cannot create new efficient models since they have limited knowledge of both hardware and framework. Auto-tuning support in TensorRT tries to solve this issue so that users create relatively tuned models. Finally, inferencing using half-precision floating numbers (FP16) is supported by almost all frameworks, similar to quantization.
This section provides an overview of our hardware platforms, including edge devices and edge accelerators, as well as FPGA-based and CPU/GPU high-performance computing (HPC) platforms. Table III summarizes their internal organization and their measured idle and average power usage.
(1) Raspberry Pi 3B: Raspberry Pi 3B [34] is a small and affordable single-board computer. On a Broadcom SoC, Raspberry Pi has a quad-core ARM Cortex-A53 processor clocking at 1.2 GHz connected to a 1 GB LPDDR2 RAM. Raspberry Pi has no GPGPU capability and no specialized hardware accelerator. Raspberry Pi and Arduino [75], with similar specifications, are good representations of IoT/Edge devices and are the main platforms for portable robots (See Table III footnotes about next generation of Raspberry Pi).
(2) Jetson TX2: Jetson TX2 [69] is a high-performance embedded platform with a 256-core Pascal architecture GPU. Jetson TX2 has two sets of CPUs, quad-core ARM Cortex-A57 processor at 2.0 GHz and dual-core Nvidia Denver 2 at 2.0 GHz. The total available memory is 8 GB LPDDR4 with a 128-bit interface. The memory is hard-wired to the memory controller and is shared between the ARM processor and GPU. Thus, the memory transfer speed of the GPU is not limited by conventional PCIe bus speeds. Jetson TX2 does not utilize hardware accelerators.
(3) Jetson Nano: Nvidia Jetson Nano [36] board is a smaller version of Jetson TX2 for edge applications. Jetson Nano is a GPU-based single-board computer with a 128-core Maxwell architecture GPU and a quad-core ARM Cortex-A57 clocking at 1.43 GHz. The total available memory is 4 GB LPDDR4 with a 64-bit interface. Similar to Jetson TX2, Jetson Nano's memory is shared between the processor and GPU. (4) EdgeTPU: Edge TPU [35] is a small ASIC designed by Google that provides high-performance DNN acceleration for low-power and edge devices. The ASIC design is hosted on a single-board computer, similar to Raspberry Pi. With an NXP SoC, the host computer has a quad-core ARM Cortex-A53 processor and one Cortex-M4 processor. The available memory for the processors is 1 GB LPDDR4. The design of EdgeTPU and its specifications have not been released.
(5) Movidius Neural Compute Stick: Intel's Movidius Neural Compute Stick (NCS) [37] is a plug-and-play USB device that connects to any platform for accelerating DNNs. It is based on the Intel Movidius Myriad 2 vision processing unit (VPU), which has 12 Streaming Hybrid Architecture Vector Engine (SHAVE) Cores. The internal core architecture of SHAVE is VLIW, with single instruction multiple data (SIMD) functional units. VPU natively supports mixed precisions, 32-, 16-, and some 8-bit datatypes. NCS was among the first special hardware designs for edge devices. (See Table III footnotes about next NCS). (6) PYNQ: PYNQ [64] board is an open-source endeavor from Xilinx that provides SoCs integrated with FPGA fabric (Zynq series). PYNQ is designed for embedded applications. We use the PYQN-Z1 board that has a dual-core Cortex-A9 processor at 650 MHz and a 512 MB DDR3 memory with a 16-bit interface. The FPGA is from the Artix-7 family with 13,300 logic slices, 220 DSP slices, and 630 KB BRAM. (7) HPC Platforms: As a point of comparison with the lightweight edge devices and accelerators, Table III also lists the characteristics and power usage of a Xeon CPU and three GPUs that are known platforms for running DNN computations in the cloud and on servers.
Execution Time: For each experiment, the execution time is measured by running several single-batch inferences in a loop.
To accurately measure time per inference as an end user, we do not include any initialization time (e.g., library loading, live input setup, and model weight loading) because this is a one-time cost that occurs during device setup. For frameworks that permit us to bypass initialization time in their code, we time their inferences by only accounting for inference time. For other frameworks, we run single-batch inferences several times (200-1000) to reduce the impact of initialization. Power Measurements: For devices, the power of which is supplied through USB ports, we measure power using a USB digital multimeter [76] that records voltage and current every second. The accuracy of the multimeter for voltage and current are ±(0.05% + 2digits) and ±(0.1% + 4digits), respectively. For devices, the power of which is supplied with an outlet, we use a power analyzer, with an accuracy of ±0.005 W. Thermal Measurements: For thermal evaluations, each experiment runs until the temperature reaches steady-state in the room temperature. We measure the processor surface temperature of each device using a thermal camera, Flir One [77]. For devices with heatsink (see Table VI), the measured temperature is the surface temperature of the heatsink. Since the thermal resistance of a transistor chip is smaller than that of the heatsink, the temperature of the heatsink surface is 5-10 degrees Celsius lower than that of the in-package junction [78].
This section studies the characteristics of edge devices and frameworks for DNN inference. Table IV summarizes all experiments and refers to the related sub-sections and figures.
To choose a well-performing framework for each device (see Section VI-B for cross-framework analysis), this section evaluates them by measuring the inference time of several models. Before analyzing the results, Table V summarizes the compatibility of models and platforms. For instance, on RPi, we deployed all models, but larger models required a dynamic computation graph because of the small memory size of RPi. Such models experience an order of magnitude higher inference time, marked with (i.e., AlexNet, VGG16, and C3D). In these scenarios, PyTorch uses its dynamic graph to manage limited memory availability, whereas TensorFlow fails to run such models. For smaller models, however, TensorFlow achieves better performance than PyTorch. Moreover, as Table V lists, we were unable to run a few models on some platforms. As an example, for the SSD model on RPi and C3D on Movidius, marked with , we face several code incompatibility issues in the base code implementation -SSD uses an extra image processing library on top of DNN frameworks. On EdgeTPU, several reasons prevented us from converting models to TFLite, marked with . This is because (i) only the quantized model is supported by the edgeTPU compiler; (ii) in some cases, the quantized model can only be obtained by a quantization-aware training, not post-training quantization; (iii) quantization-aware training or freezing the graph in TensorFlow does not necessarily create compatible quantized models for the EdgeTPU compilation tool; and (iv) obtaining the compatible TFLite model is possible with careful fine-tuning, but we were unable to find such parameters 4 . For PYQN board experiments, although the frameworks we use (FINN and TVM) have implemented small models (CifarNet, and ResNet-18), we face challenges in extending their framework to larger models. This is because of limited resources on the FPGA for larger models (for TVM VTA) and retraining requirements (for FINN). Additionally, not every model currently complies to VTA compatible code, since some parameters of the model must match the hardware specification [79]. Besides, large models require runtime data transfer between the host memory and the FPGA, which causes severe slowdowns. 5 Figure 2 illustrates time per inference in milliseconds (ms) for several models on edge devices. Time per inference per model varies widely across devices (Figure 2). In most cases, either GPU-based devices or EdgeTPU provides the best performance. However, as shown in Table V, EdgeTPU faces several barriers in compiling new models to TFLite. Although this is an engineering problem, it limits EdgeTPU end-user usability, especially for new models. On the other hand, the models in Jetson Nano/Jetson TX work out of the shelf with automatic tuning. For Jetson Nano, TensorRT yields the best performance numbers by applying several optimizations (see Table II), whereas, for Jetson TX2, most of the results are with PyTorch with no optimization. As shown, the Jetson TX2 4 Although it is a common practice to use randomized weights for the performance evaluation of a model, such evaluations are not close to reality. This is because TFLite uses quantized weights. So, the final performance is directly proportional to the sparsity of weights and their sparsity pattern. So, although using randomized weights is a proxy/shortcut to the final performance number, it would not be realistic data. 5 After the paper acceptance, based on reviewers' comments, we tried to add some data points for additional small models for the PYNQ board. However, first, we could not find a suitable frontend that could compile the models without tight hardware/software constraints. Regularly, we had to change a model to be able to execute it on the board, which required training. Second, even after changing the model, we found that a nonoptimized hardware implementation could be slower than its CPU-based implementations. Therefore, for optimizing the hardware implementation, one must profile several implementations to tune hardware parameters, a timeconsuming task. Nevertheless, we believe the endeavors to bridge the gap between software and hardware for DNNs are instrumental.
Edge computing with advantages such as high performance, security, and scalability has introduced a new paradigm for processing DNNs. To enable the inference of DNNs at the edge, a group of studies has proposed techniques that trade accuracy (sometimes) with performance to create small models. Pruning [18], [23]- [25], [72], [73] is one example of common practices that remove the close-to-zero weights. Other examples of the first group of efforts are quantization and low-precision inference [26]- [29], [83], which modify the representations of numbers to achieve simpler calculations. The second group of studies develops mobile-specific models [40], [84]- [87]. They aim to handcraft efficient operations or models to reduce the number of parameters [84] to create efficient operation to minimize computation density [40], or use resource-efficient connections [87]. The third group of studies distributes the computations of models among available resource-constrained edge devices. For instance, Neurosurgeon [88] dynamically partitions a DNN model between a single edge device and the cloud. MoDNN [89] creates a local distributed mobile computing system and accelerates DNN computations with model parallelism. Hadidi et al. [11], [90]- [92] investigate the distribution of DNN models for single-batch inferences with model-parallelism methods, while deploying distributed systems in robots [14], [93], IoT devices [94], and FPGAs [79].
The importance of in-the-edge inference of DNNs that has been shown in the preceding work has encouraged researchers and industry to propose several lightweight frameworks and specialized edge devices. Selecting an appropriate combination of framework and hardware platform for a specific application requires characterization. The following are few studies that have some of the edge and devices. In the first characterization study of edge devices [95], the inference time and energy consumption of five well-known CNN models are measured on NCS, Intel Joule 570X, and Raspberry Pi 3B, in which TensorFlow, Caffe, NCSDK, and OpenBlas are used as frameworks. The paper suggests that Raspberry Pi with Caffe requires the least amount of power.
Another study [96] evaluates the latency, memory footprint, and energy of using TensorFlow, Caffe2, MXNet, PyTorch, and TensorFlow Lite for the inference of two CNNs on MacBook Pro, Intel's Fog Reference Design, Jetson TX2, Raspberry Pi 3B+, and Huawei Nexus 6P (Nexus 6P). This research concludes that TensorFlow runs larger models faster than Caffe2 does, and vice versa for smaller models. Moreover, they suggest that PyTorch is more memory efficient than other frameworks. The latency and throughput of CNN inference are studied in another work [97], in which TensorFlow and TensorRT are frameworks to target Jetson TX2 as a mobile device in comparison with CPU and GPU platforms. Besides the preceding real-world characterization studies, SyNERGY [98] extends the performance evaluation of DNNs by proposing a fine-grained energy prediction framework on edge platforms. SyNERGY integrates ARM Streamline Performance Analyzer with the Caffe and CuDNNv5 frameworks to quantify the energy consumption of DNNs on the Nvidia Jetson TX1. MLModelScope [99], an ongoing endeavor, tries to explore the accuracy and performance of the models across frameworks, models, and systems.
In addition to the frameworks studied in this work and prior characterization papers, Apache MXNet [100] and TVM [66] used in Amazon Web Services, and Microsoft Cognitive Toolkit (CNTK) [101], utilized in Microsoft Azure, are other three open-sourced DNN frameworks backed by industry. MXNet is scalable to support distribution on the dynamic cloud infrastructure and is flexible to support imperative and symbolic programming. TVM implemented extensive compiler-level optimizations. CNTK, which implements DNNs as a series of computational steps via a directed graph, provides advantages such as combining well-known neural network types (e.g., feed-forward DNNs, CNNs, RNNs, and LSTMs). Our experiments can be extended to these frameworks by running our provided source code in our GitHub.
Finally, several recent academic efforts have proposed custom accelerators [73], [102], [102]- [111], which improve inference by utilizing sparsity, reducing memory accesses, or employing efficient dataflows. In addition, many of the custom designs have targeted FPGA/ASIC implementations for inference acceleration [79], [112]- [117].
This paper investigated the in-the-edge inference of DNNs from the perspectives of execution time, energy consumption, and temperature. We hope that the following insights from our research lead users to knowingly choose their required package (i.e., a combination of framework and platform) for a specific edge application. Additionally, we observed that even though DNN computations are heavily dominated by matrix-matrix multiplications, custom hardware designs for edge devices matters in final performance. We sought to give a perspective into the performance of edge devices with our experiments that covered several technologies (hardware with different microarchitecture/circuit designs), software frameworks, and optimization techniques. We noted hardware and software codesigns and their support on edge devices to be one of the main reasons behind achieving low inference time. Our energy measurements showed a tradeoff between energy consumption and inference time on edge devices (e.g., Movidius vs. Jetson Nano). We believe that such a tradeoff could be utilized to design efficient and application-specific devices. Finally, we analyzed the crucial impact of DNN frameworks by studying their software stack, optimizations, and virtualization.
More info at comparch.gatech.edu/hparch/edgeBench.
Please note that these frameworks are actively being developed. As a result of that, although the technical statements about each framework might change, the discussions about the mentioned optimizations still hold true.