INTRODUCTION

Deep Neural Networks (DNNs) used for machine learning (ML) and inference have been continually improving in accuracy, while becoming more complex and memory and computation resource-hungry. Executing DNNs on Graphics Processing Units (GPUs) has become key to meet quality of experience and service level objectives (SLO) [12,42].

DNN models used for inference services inherently have a rich amount of parallelism. However, even the most complex of the DNN models (VGG-19 [53] with 20 giga oating point operations (GFLOPS), GNMT [58] with 210 million parameters [14]) are unable to fully exploit the potential of massive compute and memory-bandwidth capabilities offered by the modern GPUs. Especially the discrete-GPUs (d-GPUs) e.g., NVIDIA Tesla V100 oers ⇠125 Tera oating point operations per second (TFLOPS), and 900GBps memory bandwidth) [45]. As such, time-sharing the GPU to execute dierent DNN models often leads to wastage of GPU resources [13,46,62]. Hence, it is desirable to spatially multiplex the GPU across multiple DNNs and accordingly allocate just the right amount of GPU resources to boost overall throughput without sacricing the individual model's inference performance. This requires us to understand the limits of parallelism that a particular DNN model can exploit.

The state-of-the-art CUDA MPS (multi-process service) helps improve GPU utilization by enabling spatial sharing of the GPU across multiple processes. But the default mode of MPS oversubscribes the GPU 1 resulting in uncontrolled spatial sharing. Thus, it cannot guarantee performance isolation and often results in unpredictable application throughput and latency [34]. Multiple streams and large batches further exacerbate this unpredictability with MPS [60]. As an alternative to its default operation, MPS also allows us to set a xed limit for a process's GPU %, with the goal of isolating the performance of dierent processes. However, this static provisioning means the MPS provisioning would not be adaptive to workload variations. When the workload changes, that statically xed GPU% can result in over-provisioning (thus wasting the GPU) or under-provisioning (hurting application GSLICE includes several optimizations to improve GPU utilization and loading time of IF models to the GPU ( §3.2.4).

We recognize 'Batching' improves throughput and GPU utilization for inference applications, but impact latency. Dynamically adapting the batch size is key in achieving improved throughput while keeping latency below the SLO. We devise a self-learning, adaptive batching scheme that factors the network, CPU and GPU processing costs, and SLO constraints, to batch just the sucient number of requests (e.g., images) for inference. ( §3.3.1) GSLICE also supports zero-copy data transfer to the GPU. We share DPDK's page-locked memory with GPU and leverage the GPU's DMA to directly scatter-gather data from the network packets. ( §3.3.2). Many popular DNN models have large memory requirements for storing the model weights and parameters. They account for ⇠10-30% of total GPU memory footprint (Table 1). Memory intensive parameters (i.e., Weights) of IF models do not change across inference executions. Thus they can be shared and reused across multiple instances. Hence, we implement a method for parameter sharing across multiple IFs once transferred to GPU memory. This allows us to drastically reduce the IF's memory footprint on the GPU, and allows for scaling and multiplexing a larger number of IFs on memory constrained GPU devices. ( §3.3.3). Our work complements the optimizations proposed in works [16,28,32] to lower the GPU memory footprint for IF models.

BACKGROUND & MOTIVATION

We study several widely used ML frameworks and models, and share experimental observations with spatial sharing and the limited use of GPU resources by dierent IF models.

Maximizing GPU Utilization

We briey describe the key relevant approaches proposed to improve performance and GPU utilization.

2.1.1 Batched execution. Batch processing improves GPU utilization because: i) the GPU can spawn additional compute threads to work on multiple requests in parallel; ii) helps amortize the CPU-GPU interaction and GPU transaction overheads [27]. However, it can result in high latency, which can be undesirable. We propose adaptive batching schemes, to reap the benets of batch processing, while controlling the latency (see section §3.3.1).

2.1.2 Spatial GPU Multiplexing. NVIDIA GPUs can be spatially shared using CUDA streams and MPS to provide concurrent execution of GPU kernels within and across multiple processes respectively. Streams allow launching and execution of kernels concurrently within a single process [47]. CUDA MPS enables spatial sharing of the GPU and allows concurrent execution of kernels from dierent processes. By default, an MPS client can use all the available threads (i.e., 100% of GPU resources). However, this results in a large GPU memory footprint for the process, which is wasteful when the GPU is shared across multiple clients. Further, GPU resource contention among kernels from dierent processes can result in unpredictable latency for applications.

We experimented with three dierent IFs (IF-1 and IF-2 are ResNet-50 and IF-3 is VGG-19). We start with IF-1 and after 10 seconds(s) we launch IF-2 and bring up IF-3 after 20s. We repeat the same experiment for dierent scenarios a) without MPS. b) Default MPS. c) explicit allocation of GPU%. Fig. 1 shows the impact on throughput and latency for these scenarios. With the baseline, temporal sharing of GPU 'No MPS' case, both the latency and throughput degrade with every new addition of an IF. The latency of heavier model (VGG-19) rises from ~20 ms with 1 model to ~60 ms when all 3 models are running concurrently. Moreover, the rst instance of ResNet-50's latency also rises from ~15 ms to higher than 20 ms. With temporal sharing, adding a model on the GPU causes the latency to rise for all concurrently running models. With 'Default MPS' mode, we can observe that uncontrolled spatial sharing results in the heavier VGG-19 getting the major share of GPU and getting better throughput than with 'No MPS'. But, the two lighter ResNet-50 instances see degraded performance (both in throughput and latency), with latency rising from ~10 ms to ~30 ms due to GPU resource contention. Throughput drops from ~100 images/sec to ~30 images/sec. Hence, in both cases ('No MPS' and 'Default MPS'), the interference due to addition/removal of IFs results in unpredictable throughput and latency.

MPS with resource provisioning: On the NVIDIA Volta architecture, CUDA MPS allows limiting the available GPU threads (in units of Streaming Multiprocessors (SMs)) for each process. This helps us improve the GPU utilization by apportioning just the right amount of GPU resources, while at the same time providing the benets of performance isolation and reduced GPU memory footprint [46]. While this feature is useful, it still requires more careful consideration and resource management to properly reap the benets, while fully utilizing the GPU. We repeat the previous experiments with xed resource provisioning. We explicitly restrict each IFs to use only a xed portion of available GPU resources. We limit the heavy VGG-19 IF to 50% GPU, and allocate the two ResNet-50 IFs with 25% GPU each. This results in almost interference-free execution of the IFs (when all 3 IFs are multiplexed). Further, it outperforms the other two modes, providing almost 50% higher aggregate throughput. Note that when only a single function (IF-1), or two functions (IF-1 and IF-2) are executing (between 0 and 15sec), the throughput is lower than the 'Default MPS' and 'no MPS' modes. This is primarily due to our limiting the GPU resources for these two instances. This demonstrates that, although xed resource allocation and isolation is helpful, it has to be further improved to achieve good system performance. Thus, along with MPS, a judicious and dynamic resource allocation for the contending IFs is vital to maximize GPU utilization. We should note that using GSLICE's dynamic provisioning of dierent GPU% does not aect results of DNN operation as the exact same DNN kernels are used to infer requests.

We also conducted experiments with dierent combinations of models. Due to lack of space, we omit these results. Interestingly, we observed that the 'No MPS' case always provided an equal share (round-robin time-sliced scheduling). However, the 'Default MPS' mode preferred the heavier models (due to uncontrolled spatial sharing), resulting in performance degradation of the light-weight models. We also ran experiments to see if running multiple models concurrently with resource isolation will cause an impact on throughput or latency due to contention for other GPU resources such as memory. We observed the same average inference latency for a model running at a xed GPU%, whether a single instance of a model was running in the GPU, or the model was concurrently running with other models. We nd this to be true, as long as the contention from other models doesn't bring the GPU% allocated to this model below its "knee".

2.1.3 Identifying the operating point for IF models. We proled several dierent types of IF models, viz. Image (VGG-16, VGG-19, ResNet-50), Audio (Jasper) and text (GNMTv2) to understand their limits on exploiting parallelism and to derive the kneepoint for an IF in terms of the amount of GPU resources that achieve the best balance between achieved performance and the GPU resources expended. Figures 2a and2b show the throughput and latency for processing a task (batch size 1) by a variety of IF models provided with dierent GPU percentages. The rst insight we derive is that dierent models exhibit dierent degrees of parallelism. Individual DNN layers can have many parallel kernels that can concurrently execute in a GPU, exhibiting quite a bit of parallelism. However, most of the DNN layers are still unable to fully utilize the signicant parallelism oered by high-powered GPUs. Note, both the throughput and latency reach a point of diminishing return for Jasper at 50% GPU, for GNMT, VGG-19 after 70%, for Resnet-50 at ~60%. We believe this limit arises due to inherent communication overheads and non-parallelizable work performed across dierent layers of the DNN models. Therefore, it is wasteful to provide full GPU to the models (as is done when the GPU is only temporally shared). It is preferable to spatially share the GPU to eectively utilize the GPU resources. Second, we can observe that provisioning the GPU% has signicant impact on both throughput and latency; and the relationship is non-linear. It is crucial to nd the right GPU% that would allows us to meet the demand (arrival rate of requests) while being within the budgeted latency (SLO) for each IF. At the same time, it is also necessary to maximize the aggregate throughput for all concurrently executing IFs and balance the overall GPU resource demand across the contending IFs. This requires a mechanism to apportion the GPU% to each of the concurrently executing inference applications, as a function of their request arrival rate, inference computation cost and SLO. Batch Size & Resource Provisioning Dependency Further, to understand the implications of batched execution of tasks on CUDA MPS resource provisioning, we extensively evaluated dierent ML models (Alexnet, ResNet, VGG, etc. ) on both CNTK and TensorRT frameworks for dierent xed batch sizes. Figure 3 shows the impact of batch size when varying the GPU% for a particular Alexnet model. We observe that larger batch sizes increase both throughput and latency. Also, for a given batch size, increasing the GPU% increases the throughput (Fig. 3a) until it reaches a point of diminishing returns. e.g., for a batch size of 1, we clearly observe that a GPU% beyond 30% ('kneepoint') results in only a marginal improvement, and the corresponding latency (Fig. 3b) reduction is also minimal. Moreover, we can observe that with larger batch sizes, the knee shifts towards larger GPU%. The results were similar for other models too. The key challenge here is to correctly identify and provision the GPU% for all the IFs.

GPU-based IF System Design

Challenges. We summarize the major challenges in building a scalable spatially shared inference platform. First, determining the correct amount of GPU resources to provision dierent IF applications based on the desired SLO (latency) and throughput is key to eective multiplexing. However, provisioning the right amount of GPU resources is complex for the following reasons: i) IF models with dierent complexity demand dierent amounts of GPU resources. ii) Operating with dierent batch sizes and concurrent streams further change the GPU resource demand. iii) Underprovisioning the GPU resources for an IF can hinder meeting that application's SLO, while overprovisioning degrades GPU utilization. Finally, we observe that being IF application-aware in sharing parameters across instances and eciently moving streaming data to the GPU subsystem is also important to maximize performance. Quantifying GPU Utilization We noted that the 'GPU occupancy' provided by CUDA is misleading and shows 100% even when only one thread is active on the GPU; and as such there isn't a good indicator for measuring the eectiveness of GPU utilization. We introduce a metric, the GPU utilization ecacy (GUE ([)) for dierent modes of operation as:

GSLICE: ARCHITECTURE & DESIGN

Fig. 4 shows the architecture of GSLICE. We describe the key components of GSLICE and their roles below.

Integrated Platform for IF

Today's general IF platforms need to support a number of heterogeneous ML frameworks (ML libraries) like CNTK [51],

Pytorch [19], TensorRT [44] etc. While they may run on the CPU alone, we focus on designing a platform that uses the GPU to provide low latency. GSLICE provides a 'C/C++' native platform to build and deploy IF functions (IFs). In order to support heterogeneous ML frameworks, we use a callback model to present a minimal set of generic interfaces that abstract out framework-specic functionality in a library (called libml), described below. This allows libml to provide basic IF services without being coupled with any particular ML framework. The IF Manager component is the primary DPDK [1] process responsible for routing and delivering network packets to libml. While the IF Manager is agnostic of the ML frameworks running, it has an IF model loader component that holds the IF's prole i.e., knee information for DNN model, CPU-GPU buer requirement etc. . libml implements a GPU resource allocation manager that determines and allocates the right amount of GPU resources to each IF. It also interacts with an Orchestrator service to restart IFs when the GPU (%) resources are readjusted. The libml provides the following features to IFs, i) zero-copy network packet data transfer and aggregation (e.g., of images), ii) the buer pools (CPU and GPU memory) for batched execution. iii) GPU data transfer (DTTx in Fig. 4), iv) streaming engine to manage multiple streams (spatial GPU sharing) for performing inference, v) adaptive batching to intelligently perform batched executions within the latency limits specied by the SLO. Further, it provides the callback APIs to the IFs to setup framework specic functions. Once IFs setup the framework specic functions, libml can perform IF services transparently.We have kept libml lightweight with just 6 interfaces (init, deinit, load_model, link_model, congure_batch, infer_batch). Fewer interfaces allows us to quickly adjust libml to work with newer version of ML frameworks, thus, keeping libml usable even with ML framework churn. The Orchestrator is responsible for instantiating new IFs and decommissioning an existing IF. We leverage ZeroMQ [30] to build asynchronous message-based communication between the IF Manager and the Orchestrator.

Key Features and Design choices

3.2.1 Lightweight monitoring . We determine the bottleneck entity (CPU or GPU) in an IF operation based on timestamps tracked for task (e.g., image) aggregation (rst packet to last packet) and inference processing time on GPU and CPU respectively. 5 libml tracks the timestamps and noties the IF Manager which takes timely actions to overcome the bottleneck. The CPU becomes the bottleneck when performing data copy operations (e.g., preparing the aggregated image 5 CPU time: ) 4=3 :(release all packets) -) BC0AC :(when the rst request arrives); GPU time: time to transfer & infer a batch of requests. by copying data from multiple packet payloads into a pinned contiguous buer). We use a zero-copy GPU scatter-gather approach (e.g., ooading image data transfer to GPU), to overcome the CPU bottleneck.

The IF Manager tracks notications from libml and readjusts the GPU resources for bottlenecked IFs. However, such a readjustment is a very expensive operation, since we have to restart the IFs in order for the newly provisioned GPU resources to take eect. This time varies, but is generally of the order of 2-15s. Therefore, the IF Manager re-provisions the GPU% only when essential, readjusting over a coarse time-interval. We track the timestamp at each IF for the last readjustment and hold-down any further changes till a specied minimum time (e.g., 10 sec.) has elapsed.

Self-tuning GPU Resources of an IF (GPU%) .

In order to ensure timely inference operation and dynamic adaptation to demand variation (arrival rate), we develop a self-tuning mechanism to enable IFs to readjust their GPU resource share. We consider the following two aspects to distinguish the correct resource provisioning for an IF. i) 'Residual Latency capacity': the dierence between SLO and observed latency for inference. ii) 'Residual Throughput capacity': the dierence between the achieved GPU throughput and the actual demand (arrival rate). When either of these residual capacities is negative, we try to proportionally increase the GPU%, and when both the capacities are positive and larger than a specied threshold (greater than 5%, to accommodate variability in latency), we try to reduce the provisioned GPU% in proportion to the available capacity. The resource allocation manager handles re-provisioning of GPU resources for all the IFs.

The pseudocode for the self-tuning GPU resource allocation is described in Algorithm 1. We determine the GPU% based on current inference latency and throughput to best match the SLO for latency and improve throughput the most (line 12/13). The output feeds into a MAX-MIN fair GPU resource allocation algorithm which computes the GPU% to IFs based on their demand, with the demand of IF's requiring lowest GPU% being fullled rst. To avoid oscillations, an IF's GPU% is reallocated only when the change is above a threshold (we set it to 5% as default).

Resource Allocation Manager. During initialization, each IF indicates the desired model to the IF Manager and

gets the portion of the GPU it can use. This communication and setting of the GPU% is completely abstracted within libml and performed transparently without any explicit involvement of the IF. The IF Manager looks up the model repository and identies the optimal GPU% ('kneepoint') for the requested model -this serves as the 'elastic demand' for the model. 6 Further, we choose a conservative value of 30% (a compile time congurable parameter) for IF models that do not have an apriori proled GPU%, and our self-tuning resource allocation allows the system to dynamically determine the appropriate GPU% based on the observed latency, throughput and arrival rate. Our resource allocation scheme provides a weighted (the weight being the newGPU% required by an IF in Algo. 1) allocation as follows:

• Resources (GPU%) are allocated to IFs based on demand, maximizing the minimum resource allocation for any IFs with unsatised demands. The unsatised IFs get resource shares in proportion to their normalized (sum of newGPU%) weights across all currently running IF models.

• No IF obtains a resource share larger than its demand, unless the GPU is underutilized. Note: Any new IF instance addition or removal triggers a resource reallocation and the resources for other active IFs are readjusted (as we show, with very little downtime).

3.2.4 IF resource adjustment with zero downtime. To meet the SLO and varying trac demands, it is necessary to reprovision the GPU resources so that we can maximize GPU utilization and IF throughput. This is non-trivial because after computing the correct GPU%, we have to restart the IF process and reapply the GPU resources for the IF 7 .

In order to amortize the long startup cost, we developed a low-cost 'shadow IF' and 'overlapped execution' mechanisms. They combine to transparently re-provision the GPU resources of an IF with a quick switchover of processing to the 'shadow IF'. Our evaluations show the mean idle time for this IF switchover is less than 100`seconds. Shadow IF: For every IF (primary/active), we also instantiate a shadow IF (hot-standby). Shadow IFs share the same resources (buer pool, packet ring buers, ML framework, CPU core) as the primary, but the key dierence is that they do not access the GPU and packet ring buers. Note: After initialization, the shadow IFs do not consume any CPU until they are explicitly transitioned to active state by the IF Manager. Thus, they have no adverse impact on the performance of the primary IFs. The shadow IF masks the instance initialization (DPDK and ML framework) time and brings down the GPU load time of IF model. However, this solves only half the problem, as the GPU and ML platform (e.g., PyTorch) initialization, still incur delays of the order of a few seconds. Therefore, we let the 'Shadow IF' perform GPU initialization and overlap its execution with active IF. Intermediate load: Depending on the type of ML framework, the shadow IFs load the IF model to CPU and immediately yield 8 . This intermediate loading of the IF model on the CPU helps to signicantly reduce the load time to GPU by exploiting the CPU cache memory and avoiding disk reads and re-serialization of the model data. Only after the IF Manager provisions the GPU resources and wakes up the shadow IF, does it continue loading the model to GPU. Overlapped Execution: Upon a GPU% readjustment, the IF Manager congures the GPU%, wakes up the shadow IF and switches its role to being the primary/active IF. This requires strict co-ordination with the previous primary/active IF to ensure correctness and avoid data corruption (network packets, which reside in common buer pool). While the shadow IF model is loaded into the GPU to become ready for execution, the previous primary/active model continues to execute. The IF Manager waits till the previous primary IF completes the current inference completely. It then handles the transitioning and switch over of roles for the primary and shadow IFs in co-ordination with libml. It switches the shadow IF to 'active' and eventually terminates the previously active IF. This nal coordination by manager incurs around 50-60`seconds (idle time). Overall, this mechanism masks the IF's startup time (order of 2-15s) and improves overall system performance.

How many CUDA Streams?

Although, CUDA Streams improve GPU utilization and overall throughput, they have an adverse impact on latency. We show through experiments with dierent batch sizes, the impact of using multiple streams. In particular, we see the results for ResNet-50 model with TensortRT in Fig. 5a. Beyond 2 streams (for batch sizes of 1,8), there is barely any improvement in throughput. But, the latency keeps increasing. We observed the same with VGG-19. We saw that Alexnet was an exception that showed improvement with more streams, but only for the small batch size (1). Our analysis is that streams are helpful only when there are sucient resources to take advantage of multiplexing. Otherwise, the tasks contend for the limited resources resulting in high execution overhead, reected by the latency increase. Hence, streams are benecial only for light weight models and lower batch sizes. However, we do see some drawbacks of using a single stream for processing. We proled for the single stream case for Alexnet, as shown in Fig. 5b. The GPU remains idle for a fairly large amount of time, ⇠700 `s, between every execution. This corresponds to the time taken to notify the CPU of inference completion and the processing on the CPU side to perform cleanup and return the callback. Note that, until the callback is completed, the CUDA driver does not launch the next inference execution, even if the tasks were queued before. Fig. 5c shows the proler results with two streams for Alexnet. We still observe similar gaps in each of the streams, but since the two streams have overlapped execution, the GPU utilization is higher in this case.

However, the streams are problematic while multiplexing models with unequal compute requirement e.g., ResNext-50 running concurrently with Alexnet. In such cases, streams do not provide any meaningful overlap in the execution of the kernels of the distinct streams, as seen in Fig. 6. Here, ResNext-50's kernels (stream 14) do not overlap with the execution of Alexnet's kernels (stream 16). Thus, the streams are restricted to time-share share the GPU, resulting in increased inference task completion time. This causes lower utilization of GPU as well, since only one application is running at a time in the GPU. We can extract higher GPU utilization by rather running each DNN as an individual application and spatially sharing the GPU using MPS with resource usage limits. Fig. 7 shows that ResNext-50 and Alexnet can concurrently run in the GPU if we run them with MPS with a resource usage limit. This allows for much better utilization of GPU as well as lower latency, as one DNN's inference does not have to wait for other's to end. Therefore, in GSLICE we prefer using MPS with resource provisioning instead of CUDA streams, whenever possible. We limit to 2 streams in the cases where streams can help boost the throughput.

3.3.1 Self-learning Adaptive Batching. A larger batch size improves throughput and GPU utilization (amortizing the interactions between CPU and GPU). A naive approach would be to batch as many requests as possible till the previous inference operation completes. This approach yields high throughput, but can result in high latency. Alternatively, we can learn and adapt (limit) the batch size to be just sucient to meet the SLO. This adaptive batching, used in Clipper and Nexus [21,52] works well when the GPU is temporally shared. However, when the GPU is spatially shared, several additional factors also contribute to latency, namely i) variation in provisioned GPU resources; ii) multiple streams that contend to spatially share GPU; iii) interference due to concurrently executing GPU tasks, with MPS. Most of these factors are dynamic in nature and can happen outside the scope of an individual IF. Therefore, we develop a 'self-learning' approach to dynamically adapt the maximum operational batch-size. Self-learning Adaptive Batching (SLAB): On the host CPU, libml tracks the inference completion time of each batch and heuristically determines the batch size that allows us to operate within the SLO. SLAB works by increasing or decreasing the batch size in proportion to the current batch size, observed latency and deviation from the specied SLO, by checking the headroom (SLO -avg. latency for the previous batch). Thus, it quickly readjusts the batch size. An IF is considered to be underprovisioned when it misses its SLO even with a batch size of 1. Instead, if the IF operates within the SLO, then the IF may be overprovisioned. libml tracks (5 successive) such occurrences and noties the IF Manager to readjust (increase/decrease) its GPU resources.

Data Aggregation and Transfer to GPU. DNN applications such as

Imagenet models require the entire image's data (and all of the batch, if batch size is > 1) to be available in a contiguous GPU memory buer before an inference is started. To detect requests that are ready, GSLICE implements a CPU side zero-copy aggregation list of network packets (i.e., store only the packet pointers in an indexed array) and tracks aggregation status for up to 64 distinct requests (as bit elds). This allows quick detection of all ready requests (images, text, etc.) to be transferred to the GPU. We buer packets until the IF operation completes. Alternatively, we could release the packets as soon as the data is copied to GPU, requiring a callback from GPU to notify copy completion. This is expensive (40-100`s). We found it has negligible benet, and sometimes detrimental because of host-side context switching to process CUDA driver callbacks. Data transfer to GPU: To eciently transfer data without taxing the host CPU, we pin the DPDK hugepages (used for network packets) with CUDA. Then, using NetML's approach [23] the GPU's DMA performs zero-copy scattergather of the packet data using a GPU kernel. An alternative is a NIC to GPU transfer using GPUDirect. However, GPUDirect places all ingress packets into GPU memory and the GPU kernels have to perform packet processing. This has many limitations, such as inecient packet processing in GPU [55]. Using NetML's approach [23], we only transfer data for ML processing to the GPU, leaving packet processing to the CPU. This avoids the CPU copy overhead.

Shared Inference Parameters.

The IF Manager includes a model loader component that loads the inference model related parameters only once to the GPU and allows reuse of those parameters across dierent instances of the same IF. This has two benets: i) increased multiplexing due to reduced GPU memory footprint per IF; ii) faster module loading due to reuse of already mapped GPU parameters. ML frameworks (e.g., Pytorch, CNTK), export APIs to retrieve the parameters (weights and biases) of the DNN models and their GPU addresses. The IF Manager takes advantage of these platform specic API's to build the GPU address mapping for the parameters of dierent models. It then exports these GPU addresses to dierent IF instances through the CUDA API cudaIpcGetMemHandle(). We extended the CNTK and Pytorch libraries to make them attach parameters to the specied GPU address. These changes are minimal (less than 30 lines of code). Table 1 shows the benets of parameter sharing (PS) in comparison to standalone (SA) mode of operation. Parameter sharing is both time and space ecient, enabling us to swiftly load the models to GPU with 8-10⇥ reduction in load time, and t 5%-54% more IF instances.

Implementation Details

We implemented GSLICE as a DPDK [1] based platform. Overall, implementation of GSLICE is ~2.5: lines of 'C/C++' code. We minimally enhanced the CNTK and PyTorch libraries to support parameter sharing and improve inference performance. Key extensions to CNTK and PyTorch include: • Parameter Sharing: We extend ML frameworks to attach ML parameters from a specied GPU address -helps minimize startup time and memory footprint. • Reuse of GPU I/O buers: We allocate the input and output buers only once (typically done for every inference operation), and reuse the existing GPU buers for subsequent inference operations -helps avoid expensive GPU memory alloc/dealloc for each inference operation. Figure 8 and 9 show the results for GSLICE (throughput (left) and latency (right)). For clarity, we split IFs with different complexities into two groups. The error bars indicate 95% condence interval. GSLICE achieves signicant improvement over 'default MPS' in throughput for Alexnet (⇠10⇥), and Mobilenet (⇠9⇥). VGG-19 (a computationally heavy model) shows ⇠2⇥ improvement. Improvement comes from more eective GPU utilization with GSLICE. While we observe increased latency with GSLICE and 'M+S', judicious batching limits latency to be within the specied SLO. Since only one model executes at a time here, throughput is primarily improved by batching.The incremental benet of GSLICE, beyond just our adaptive batching applied to MPS (M+S), is higher for lighter models, Alexnet, Mobilenet and VGG-16. However, GSLICE signicantly improves throughput and latency, beyond the benets from adaptive batching, with multiple heterogeneous models running concurrently (see below). GSLICE improves GPU utilization ecacy (GUE) across all IFs, compared to 'default MPS' by 5-229% (Table 2). We demonstrate the benet of GSLICE in multiplexing dierent IFs in the GPU. We use combinations of 2, 3 and 4 IFs, which run concurrently in the GPU, while achieving an SLO of 25ms with varying GPU demand/load. 10b (right). GSLICE outperforms both 'default MPS' and 'M+S' to provide ⇠6-13⇥ and 1.2-4⇥ improvement in throughput respectively, across all the combinations. We note in Fig. 10b that the throughput of VGG-19 in GSLICE (22 images/sec) is lower than in SLAB (32 images/sec). In GSLICE, an IF's share of the GPU resource is determined using a Max-min fair share algorithm to maximize overall system performance without penalizing any IF more. There is some penalty for compute heavy DNN models, compared to letting them grab all the GPU resources at the expense of other models, as in the SLAB experiment. Similarly, Table 3 shows that GSLICE provides ⇠1.7 to 9⇥ improvement in GPU Utilization ecacy. Thus, GSLICE provides improved, consistent throughput and performance isolation for each IF by a judicious allocation of a xed GPU% (refer §3.2.3).

Concurrent execution of multiple IFs

Comparison with TensorRT Server

We focus on single GPU to present subsequent benets and use case demonstrations for simplicity. We compare GSLICE with state-of-the-art NVIDIA TensorRT (Triton) Server ver.19.02 [44]. For this experiment, we used the performance evaluation tool published by NVIDIA [3]. For a fair comparison, only for this experiment, we restrict GSLICE's adaptive batching to not exceed a congured maximum batch size. Note: the throughput and latency shown are purely for the execution time of a ready batch on the GPU. Thus, we eliminate any dierences as a result of the overheads due to HTTP/TCP in TensorRT and UDP processing in GSLICE. From Fig. 11a, we observe that GSLICE achieves higher throughput across all batch sizes. for large batches (4.. 16), is able to achieve 31-92% throughput than TensorRT server and 23-45% lower latency for larger batches. Also, in Fig. 11b, GSLICE outperforms TensorRT server in providing low-latency service for most batch sizes. Table 4 summarizes the performance improvements of GSLICE. Especially, with the largest batch size (16), GSLICE provides almost 2x improvement in throughput with about half the latency of TensorRT server. TensorRT server provides slightly lower latency for batch size of 1, but has much lower throughput than GSLICE. Note that although we restrict batch size in both cases, GSLICE incorporates adaptive-batching and two streams that seek to maximize GPU occupancy and minimize idling of GPU (i.e., avoids wait for a batch to complete), which is evident from the lower latency with GSLICE. In addition, Zero-Copy GPU scatter-gather ooads the CPU to further reduce the batching overhead, resulting in overall better throughput.

Benet from each GSLICE component

To illustrate the eectiveness of GSLICE's proposed techniques, we run the same experiment as in §4.1 and enable each component of GSLICE individually, viz., multi-streams, Zero-Copy GPU scatter-gather and adaptive-batching. We set the SLO to a nominal value of 25 ms. Figs. 12a and12b show the throughput and median latency respectively. With multi-streams (we limit to using only 2 CUDA streams) alone, the lighter models Alexnet (39%) and Resnet-50 (66%) show throughput improvement. With the heavier model VGG-19, as the computation load dominates, the throughput contribution from streams support is limited. But, multistreams does increase latency. Nonetheless, unlike single stream, multiple streams avoid idling the GPU during the execution of the GPU callback.

The zero-copy approach helps improve both throughput and latency, although it depends on the complexity of the IF model. For Alexnet, the data transfer takes about 100 `s, which is a signicant percentage of the inference time of ~1 millisecond. We see about a 17% increase in Alexnet's throughput (Default MPS: 460 images/sec, Zero-Copy GPU: 540 images/sec) in Fig. 12a, and the latency of inference also decreases slightly. However, the improvements are marginal for more compute intensive models like ResNet-50 and VGG-19, which spend more time (~10-15ms) for DNN computation, dwarng the data movement time. Nonetheless, the main benet of Zero-Copy GPU scatter-gather is ooading the CPU, avoiding it from becoming the bottleneck.

We note that batching a number of inference tasks improves throughput. But, this is only up-to a point where all SMs of a GPU are used. Forming and inferring larger batch sizes hurts inference latency, which can be a concern for realtime operations. Table 5 shows the impact of batching on throughput and latency for ResNet-50 and VGG-19 models on our testbed. We can see the throughput improvement is much higher from Batch 8 to 16. However, from 16 to 32 the throughput improvement is smaller, but the latency increases quickly for both models. Thus, to balance between throughput and latency, we utilize adaptive batching in GSLICE.

Adaptive batching helps improve throughput for all models. In fact, the heavier the model (e.g., VGG-19) the larger the improvement. With lighter models (e.g., Alexnet, ResNet-50),

SLO (ms)

Clipper Thpt. GSLICE Thpt.

Clipper CT GSLICE CT

Figure 14: GSLICE vs Clipper: for dierent SLOs. the improvement is still good, as shown in Fig. 12a. However, we do observe an increase in latency (2-3x) for all the models compared to the baseline MPS with each of our improvements, especially with VGG-19 and adaptive batching. We note that although the latency is higher than default MPS, we can still maintain an SLO of 25ms. GSLICE is able to use all the available time-budget to right-size the batch, maximizing throughput while limiting the latency increase.

Self-learning Adaptive Batching

We compare GSLICE's adaptive batching with another inference system, Clipper [21]. We integrated the adaptive batching code from Clipper's Github repository [4]. We use a VGG-19 on TensorRT IF (with latency ~14ms for a batch size of 1) and vary the SLO across dierent runs.

Fig. 12c shows the convergence time of the two adaptive batching schemes for SLO of 50ms. We chose a higher SLO of 50 ms with IF running VGG-19 to give it more time to form a bigger batch. VGG-19 being a compute heavy model would only form very small batch with SLO of 25ms. GSLICE uses the available latency headroom to rapidly increase the batch size (in less than 300ms) to the maximum that would not violate the SLO. On the other hand, Clipper's batch size increases gradually and takes ~3.5s to reach the right value. With Clipper, the convergence time keeps rising with higher SLOs, and exceeds 10s for SLO of 100ms, while GSLICE quickly converges within 500ms in all the cases as shown in Fig. 14 (right).

We also show the impact on throughput for dierent values of the specied SLO as shown in Fig. 14 (left). GSLICE shows ~2 to 50% throughput improvement across the measured cases. At lower SLOs (25-50ms), we observed Clipper to be sensitive to SLO violations (due to multiplicative decrease) and converge to a relatively smaller batch size, hurting throughput. However, GSLICE nds the right batch size to operate within the SLO and provide higher throughput.

4.6 GPU resource re-provisioning 4.6.1 Adaptation to workload variation. To show GSLICE's ability to adapt to changing workload, we dynamically vary the workload presented to the GPU. We start with a VGG-16 IF active with an SLO of 50 ms. After 20s, we launch another VGG-19 IF. We run both (VGG-16 & VGG-19) IFs together and terminate the rst IF (VGG-16) at 50s. Figure 15, shows the timeline of events, the observed batch size, throughput, and latency for each execution round of the two IFs.

With SLAB using default MPS (both IFs are provided with 100% GPU), shown in Fig. 15a, VGG-19 IF gets a throughput of ~550 at the start (and remains below the SLO of 50 ms). The IF's throughput drops to ~200 when VGG-19 IF starts at 23-sec mark. When the VGG-16 IF stops at 53 sec, the batch size and throughput of VGG-19 IF increases quickly.foot_7 Thus, adaptive batching in SLAB adjusts the batch size to utilize freed up GPU resources.

In Fig. 15b we further demonstrate GSLICE's capability to provide proper GPU isolation. We run the same experiment with GSLICE, which provides 100% GPU to rst IF (VGG-16). We maintain the SLO of 50 ms and high throughput. When the VGG-19 IF starts, it restricts the GPU resources for VGG-19, to 30%, and provides the remaining 70% GPU allocation to VGG-16. We can see from Fig. 15b, that throughput of VGG-16 still remains quite high (~300) compared to the default MPS case even when the VGG-19 IF comes up. Once, the VGG-16 IF stops at 53 sec mark, GSLICE starts the process to increase the GPU% for VGG-19 IF by getting the Shadow VGG-19 IF ready with 100% GPU. The process of switching to a dierent GPU% is not instantaneous, as the shadow IF takes some time (~12 sec) to load the ML model into the GPU. However, the active 'primary' VGG-19 IF continues to infer new requests, with minimal downtime. When the shadow IF is ready to execute, at 66 sec., GSLICE almost Finally, we compare with Clipper in Fig. 15c, which gets an initial throughput of ~500 with VGG-16 at a latency below the SLO of 50ms while running alone in the GPU. When the VGG-19 IF starts at ~23 secs., the VGG-16 model suers from interference and struggles to maintain the SLO, and throughput drops to ~200 images per sec. When, VGG-16 stops at 53 sec., VGG-19's throughput increases slightly as more GPU resources are freed up for it. However, Clipper's batch size remains the same and the throughput remains at almost the same low value. This is due to the fact that Clipper's adaptive batching system is only reactive to violating the SLO, but once stabilized at a value, does not readjust to capitalize on now free resources, unlike GSLICE's continual adaptation. 4.6.2 Self-Tuning of GPU proportion Allocation. We extend the previous experiment ( §4.6.1) to demonstrate GSLICE's ability to re-allocate GPU resources for the active IFs in a selftuning manner, adapting to the demand without impacting their performance. We assume the 'knee' GPU percentages of the IF models are unknown, with no apriori proling. We start with two models, VGG-

RELATED WORK

GPU based Inference Platforms: Clipper [21] is a lowlatency inference system providing a common model abstraction API, and predictive IF model & ML framework selection service to better suit user requests. It uses an AIMD-based adaptive batching scheme and delayed batching modes to optimize the batch size to meet a target SLO. GSLICE goes beyond Clipper to support heterogeneous ML frameworks and models to run concurrently by spatially sharing the GPU. GSLICE also provides a variant of adaptive batching that can swiftly adapt to GPU resource variations.

Nexus [52] and Themis [41] create an inference service with a cluster-wide GPU resource management framework. Nexus proposes batching-aware scheduling using prex adaptive batching, and early drop mechanisms to improve overall performance and utilization for a GPU cluster. Nexus scheduler relies on temporal-only sharing of GPUs, with the adaptive batching operating on a xed batch size for each epoch (30s). Themis proposes a two-level GPU scheduling approach with xed batch size and a novel fairness measure called 'nish-time fair allocation' for the GPU cluster. Both Nexus and Themis operate at a higher management plane (cluster level management), while GSLICE spatially shares a GPU to maximize GPU utilization and improve IF application performance. Gandiva [59] is a DNN learning and inference architecture that increases throughput by 'packing' DNN applications in the GPU. Packing in Gandiva corresponds to temporally multiplexing applications. Temporal sharing increases utilization but not to the extent of spatial sharing. GSLICE uses controlled-spatial-multiplexing to share the GPU with applications and increase GPU utilization.

TensorRT [57] by NVIDIA Inc. is a high-performance deep learning inference library for production environments [26]. NVIDIA also released the container runtime for Docker with TensorRT support [2]. However, the current release does not yet fully support MPS. We used the publicly available TensorRT libraries and built the TensorRT IF on our platform with MPS support. PowerAI Vision Inference Server [33] by IBM Inc. is a new generation of video/image analysis platforms that can deploy both the image classication models and object detection models. EdgeEye [40] presents an edgecomputing framework for real-time intelligent video analytics applications. It uses NVIDIA TensortRT inference engines to improve inference performance, and provides a high-level, task-specic API for developers. Unlike GSLICE, their focus is to provide a framework for real-time video analytics. Works on CUDA MPS: NVIDIA's latest Ampere GPU has Multi-Instance GPU(MIG), where the GPU resource partitioning goes beyond SMs and includes memory partitioning [7]. Ampere keeps the underlying CUDA platform same, so we expect optimization in GSLICE would be benecial in Ampere GPU as well. In [34], authors identify the performance gap issues with GPU resource sharing (temporal and spatial), and propose a dynamic space-time multiplexed scheduling to optimize the GPU inference throughput. It also tries to prioritize predictability, but proposes to micro-manage the kernels that execute on GPU, monitor the latency for each kernel execution and control the eviction of degraded task. GSLICE is simpler, providing platform level optimization mechanisms that are non-intrusive (no need to micro-manage kernels), and readily leverage and deploy available IF frameworks. ML Acceleration: Several works optimize and produce lightweight versions of the DNN models [25,43,62] to make DNNs fast and less compute intensive.SqueezeNet [32], has fewer parameters and a small model size. Others accelerate ML training and inference with binarized neural networks [20] or compress the DNN model [28]. There are hardware accelerators such as NVIDIA's tensor cores [8], Google's TPU [36] and Eyeriss [17] made to speedup DNN processing. GPU accelerator functions: Many works leverage GPUs to accelerate packet processing. PacketShader [27] utilizes GPUs to process packet headers for switching and routing, and SSLShader [35] provides high throughput SSL processing in the GPU. Similarly, NBA [37] presents an adaptive load balancer to balance the workload of network functions (NFs) running on both CPU and GPU. APUNet [25] uses integrated GPU to process packets and eliminates the data transfer over PCIe bus. G-NET [61] is a scheduling and virtualization framework to share GPU resources across multiple NFs. G-NET uses Hyper-Q to spatially share the GPU, and provisions the GPU SMs for each of the NFs. In contrast, GSLICE utilizes MPS to provision GPU. Using MPS allows GSLICE to support low level ML libraries such as cuDNN [18] and cuBLAS [5] which do not expose thread blocks information and prevent the use of thread block counting technique to provision GPU.

CONCLUSIONS

We presented GSLICE, a platform supporting cloud-based low-latency inference applications. GSLICE supports a number of ML frameworks and IF models, and specically addresses the challenges of eciently utilizing the GPU while multiplexing several concurrently running streaming IFs. GSLICE builds on top of CUDA MPS to provide controlled spatial sharing of the GPU across multiple IF models to ensure performance guarantees. GSLICE's 'self-tuning' resource allocation scheme dynamically adjusts and apportions just the right amount of GPU resources for the IFs to meet their SLO and demand. GSLICE's 'Self-Learning Adaptive Batching' helps maximize GPU utilization and IF throughput without violating latency SLOs. Using a shadow IF and an ecient overlap technique, GSLICE masks the high startup costs (2-15s) and re-provisions GPU resources and instantiates new IFs with less than 100`s downtime. GSLICE also improves IF performance and reduces the overhead on CPU through data aggregation and ecient data transfer mechanisms. Parameter sharing in GSLICE improves scalability by reducing IF memory footprint and helps multiplex multiple instances of an IF. Overall, GSLICE dramatically improves GPU multiplexing (5-54%) and utilization ecacy (up to 800%) to achieve 2-13⇥ overall IF throughput improvement.