Applications of Deep Learning (DL) models, such as image processing and speech recognition, have exponentially grown in recent years [19], resulting in useful services and products [12,28]. Despite the many benets of DL models [38], their heavy computational requirements have resulted in a signicant demand for expensive and powerhungry GPUs [11]. This immense GPU demand underscores the need to fully utilize available GPUs to amortize costs.
DL models and workloads vary signicantly in their resource usage proles. Some DL workloads, such as BERTbased training [8], have high GPU compute requirements but a small GPU memory footprint. Others, such as Whisperbased inference [30], have a larger GPU memory footprint, but a moderate GPU compute requirement. Further, training and inference workloads have distinct resource needs, complicated by dierent model architectures. Even dierent batch sizes of a given model can result in signicantly dierent resource requirements; for example, in our experiments, a Whisper-based inference workload with batch size of 16 had a 33% higher GPU compute utilization compared to the same workload with a batch size of 2. Consequently, a DL workload may not fully utilize the xed resources on a given GPU model, resulting in underutilization of expensive GPUs.
While the variability in GPU resource requirements of DL workloads creates an underutilization problem, it also lends itself to a possible solution-spatially sharing GPU resources among diverse DL workloads to increase GPU utilization. Hardware support already exists for GPU spatial sharing (see Section 2). Nonetheless, there are several challenges that make it dicult to spatially share the GPU between DL workloads: • Performance interference. Sharing a GPU between just two DL workloads can result in unpredictable performance degradation for one or both colocated workloads [18,35].
For example, when we colocated a BERT [8] training workload with an ALBERT [17] training workload, they experienced a throughput drop, relative to an exclusive run, of 40% and 37%, respectively. When the same BERT workload was colocated with a ViT [10] inference workload, the corresponding throughput drops were 19% and 3%, respectively. The much lower drop in throughput in the latter case was because the colocated Vit workload was an inference workload, and so did not require high GPU compute resources (e.g., for backpropagation). While the colocated ALBERT workload in the former case has much fewer parameters, and thus a lower memory requirement, it is a Transformer-based model, requiring signicant GPU compute resources for training. • Colocation candidates. For a target DL workload that is to be executed on a GPU, there could be several candidate DL workloads for colocation (e.g., those in the ready queue of a GPU cluster service [41]). While coarse-grained resource usage patterns of individual colocation candidates can be readily obtained using an exclusive run, this information may not be enough to accurately predict the performance degradation under colocation. • Proling overhead. A natural approach to determine which workloads to colocate is to predict their performance under colocation. Machine Learning (ML) techniques are a promising solution for such predictions. However, ML techniques typically require a large training set to make accurate predictions. For the workload colocation problem, generating multiple training samples by repeatedly running colocated workloads will require signicant time and eort. Further, the time required to train complex ML models can also impose signicant overheads. In light of the above challenges, we pose our problem statement as follows: "Given a DL workload that is to be executed, how to choose a colocated DL workload to improve GPU utilization while minimizing the performance degradation of the colocated workloads?"
There has been some recent work on predicting colocated performance under GPU spatial sharing (see Section 6). However, such works typically focus on DL training jobs with checkpointing to save states [21,39], and are thus not applicable to inference workloads, which are latency sensitive, or training workloads that may have performance constraints. Further, the proling overhead can be high when predicting colocated performance. For example, multiple colocations have to be proled online (at runtime) for each target workload to be executed [18]. There are also approaches that only rely on oine proling, but require changes to the GPU driver or scheduler for support [35]. We thus notice a gap in literature on solutions that achieve ecient GPU spatial sharing without hardware/rmware changes and without signicant proling overhead.
In this work, we present KACE, a lightweight, applicationlevel solution that leverages oine proling to predict colocation performance under spatial sharing. KACE leverages these predictions at runtime to determine the best candidate workload to colocate with a given target workload. While KACE can generally be applied to any GPU workload, we focus on DL workloads in this work. Unlike recent works that only focus on long-running training jobs (where the proling overhead is not important) [43], KACE applies to both inference and training workloads.
Using the right features, we nd that a small training set (⇠20% of the entire dataset) and a simple linear regression model, with trivial training time, provide adequate prediction accuracy for KACE to eectively colocate potentially unseen workloads at runtime. KACE eliminates the need for extensive online proling, as done in recent works [18], and instead relies on one-time, oine runs of individual workloads to extract meaningful metrics. We nd that GPU kernel metrics provide valuable information for colocation, enabling KACE to outperform system-metric-based approaches.
We implement KACE in Python and evaluate its performance by comparing it with various baselines using 7 diverse DL workloads with dierent batch sizes. Our experimental results, using PyTorch for DL runs on an NVIDIA V100 GPU with MPS support, show that KACE achieves over 90% of the colocated performance that an oine-optimal oracle policy achieves using only ⇠20% training data. Compared to recent approaches, KACE provides 11% higher performance, on average, and as high as 88% for certain workloads. Compared to rule-based approaches, KACE provides 14% higher performance, on average, and as high as 52% for certain workloads. Even in the case of unseen workloads that have not been encountered in training, KACE continues to outperform other approaches, achieving 11%-16% higher performance. The code for KACE is publicly available to promote further research and facilitate reproducibility [1].
2 Background on GPU Sharing Mechanisms GPU time-sharing is a resource management strategy where multiple jobs share a GPU over time (in distinct time slots), involving context switching between jobs. Various scheduling strategies, such as round-robin [24], can be employed to schedule jobs over time. However, minimizing the context switch overhead between jobs continues to be a practical challenge (see Section 6). Further, not all DL jobs can fully utilize the GPU resources [44], suggesting the need for GPU spatial sharing for improved GPU utilization.
GPU spatial sharing allows multiple jobs to execute concurrently on a GPU by partitioning GPU resources via various supported hardware and software mechanisms.
NVIDIA Multi-Instance GPU (MIG) partitions the GPU into independent resource instances of various sizes. MIG ensures isolation of compute and memory resources but cannot dynamically adjust instance sizes on the y; reconguring a MIG instance and checkpoint-restarting a DL job could take minutes [18]. Further, the xed resource sizes of MIG instances may lead to GPU underutilization due to a lack of exibility. For example, the A100 GPU with 40GB GPU memory only supports MIG instances with memory sizes of 5GB, 10GB, 20GB, or 40GB [26]. Also, by design, MIG ensures isolation between instances, preventing the exploitation of unused GPU resources across instances. Finally, MIG is typically available only on a handful of high-end data center GPUs [26].
NVIDIA Multi-Process Service (MPS) eliminates context switches by merging colocated CUDA contexts. Specically, MPS allocates one copy of GPU storage and schedules resources for all CUDA processes, instead of holding the context separately [6]. It also allows each process to exclusively occupy Streaming Multiprocessors (SM) via the CUDA_MPS_ACTIVE_THREAD_PERCENTAGE variable. Due to its exibility, MPS has been shown to outperform MIG [32]. Further, MPS is more readily available on several GPU models than MIG. We use MPS in this work for GPU spatial sharing. Note that the underutilization due to xed-size MIG instances can be addressed by employing MPS to run multiple DL workloads within a MIG instance; in fact, MPS is supported on top of MIG [26]. As such, our solution for MPS in this work can aid MIG deployments as well.
AMD compute-unit masking in AMD GPUs allows users to specify a set of compute units on which kernels should execute [29]. This mechanism appears similar to MPS, as both enable concurrent kernel execution with memory interference. Consequently, our MPS-based solution could be applicable to AMD GPUs equipped with compute-unit masking.
This section describes the system design of KACE. The key components of KACE are: (1) Workload proler, (2) Model trainer, and (3) Colocation predictor; these are depicted in Figure 1 along with the interactions between components.
The proler collects GPU metrics for individual workloads oine. These metrics are collected one-time only.
A common approach to evaluate GPU performance is to gather overall system metrics, such as compute and memory utilization, using, for example, nvidia_smi [25]. However, we note that the compute and memory utilization metrics only indicate whether the GPU is busy, rather than providing
7/15/24, 4:35 PM KACE_sysdesign_v2 Page 1 of 1 https://app.diagrams.net/?src=about Trainer profiler Feature set totalthread sm,mem SM_busy MEM_busy MEMCAP SM_xput MEM_xput DRAM_xput threads regs sharemem workload_xput Train Offline Training Train set Online Prediction Split train set Profile Workloads Predictor Deploy model Target workload GPU Workload Profiler Profile once Profiled features Profiled features Feature set Predict Execute ( , , 0.8) ( , , 0.5) ( , , 0.3) (Target, Candidate, Xput sum) Output Candidate workloads Feature generator actual utilization of processing units (Floating Point Process Units, Tensor cores, etc.) [43]. Thus, these metrics only provide coarse-grained estimates of performance.
Kernel metrics provide detailed insights into individual kernels, including metrics such as Stream Multiprocessor (SM) throughput and achieved occupancy. These metrics oer precise performance data at the kernel level, unlike nvidia_smi metrics which provide broader system-level information. For example, SM throughput measures the maximum throughput across 22 processing pipelines, considering operations in FP64, ALU, and FMA [27].
Despite the crucial information (actual resource utilization) that kernel metrics provide for interference prediction, proling kernel metrics can take substantial time [35]. However, the proling in KACE is done oine, and is thus not on the critical path when selecting the colocation candidate at runtime (see Section 3.3). We thus leverage both system and kernel metrics for KACE. We also include individual workload throughput (obtained via an exclusive run without colocation) as a metric as it is a strong performance indicator and can be collected with system metrics without additional proling. The selected metrics we employ in KACE are: SM busy rate, memory busy rate, memory capacity, compute (SM) throughput, memory throughput, DRAM throughput, number of threads, number of registers, static shared memory, and throughput of single workload without colocation (shown, abbreviated, in Figure 1).
To collect metrics, we perform two oine prole runs for each workload (to prevent kernel proling overhead from impacting system metrics). System metrics require a short run (taking, on average, ⇠1 minute); we execute 100 steps for each workload, excluding the rst 10 steps to account for the bootstrap eect of PyTorch's lazy initialization. For the kernel metric run, we only require one step of each workload. After collecting kernel proles, we sum up the kernel metrics across kernels and weigh them by kernel duration.
To train (and test) a model for predicting the performance of colocated workloads, we run experiments where pairs of workloads are colocated. The system design and methodology of KACE can be extended beyond two colocated workloads (see Section 5.2.5), including for non-DL workloads, since the key hurdles we have to overcome to improve colocation performance are not specic to workloads pairs or DL workloads. To fully evaluate the impact of interference, we focus on performance during the interval when both workloads are executing; as such, when any one workload completes execution, we terminate our proling processes.
For the training data feature set, we combine the features of colocated workloads; we sum up the raw metrics (e.g., number of threads) and average out the metrics that are reported as percentages (e.g., busy rate). Finally, we normalize all features before training. While the trainer can accommodate any prediction model, we evaluate four dierent ML techniques in Section 5.2.
A key contribution of KACE is its ability to train with a small sample size. We evaluate prediction performance under dierent training set sizes, including the challenging scenario where the target workload to be colocated is not part of training, in Section 5.1; we nd that KACE can predict adequately using only 20%-30% of the data for training.
The predictor determines, at runtime, the workload to colocate with a given target workload to maximize performance. While any denition of performance can be employed, we consider throughput sum to be our metric in this paper; see Section 4 for details. Given a target workload to be executed, the predictor iterates through all colocation candidate workloads (e.g., from a ready queue [41]) and predicts the throughput sum for each combination of target and candidate workloads. The candidate workload predicted to provide the highest throughput sum is selected for colocation with the target workload (see Figure 1).
The KACE implementation is done at the application level, without any OS or hardware modications. We implement the core logic of KACE in Python with around 2,500 lines of code. For system metric proling, we utilize nvidia_smi pmon to capture the resource usage of processes [25]. For oine kernel proling, we use NsightCompute [27];
NVIDIA_MPS is disabled in this stage. During online prediction runs, we initiate the MPS daemon. Once the predictor determines the ideal workload for colocation, it submits them to the running MPS server for colocated execution.
Evaluation setup. We conduct our evaluation on a server in the Chameleon Cloud [16] equipped with 2 Intel Xeon Gold 6230 CPUs, 128GB RAM, and a 32GB NVIDIA V100 GPU. We use PyTorch 1.13 and CUDA12.3 for our experiments. Evaluation methodology. We consider a target workload that is scheduled for execution and a list of candidate workloads that can be colocated with the target workload. Our performance metric, for a given workload colocation, is the throughput sum of the colocated workloads, -1 + -2 , where -1 and -2 are the throughput of the target and candidate workloads, respectively, when colocated. We focus on throughput sum as it is commonly used in colocation evaluations [35]. When reporting our results, we normalize the observed throughput sum with the throughput sum achieved by Oracle, -Oracle , which is an oine-optimal policy representing the best colocated pair (described below).
Each colocation experiment is repeated 5 times. We report the average value of the normalized throughput sum along with error bars that represent the standard deviation. Workloads. We employ 21 workloads, encompassing 7 training and inference models with 3 batch sizes each. The workloads have relatively high GPU utilization, with an average SM busy rate of over 60%. Detailed workload information is shown in Table 1. By considering all possible pairs of workloads that can be accommodated in GPU memory, we obtain 181 possible colocations. We also conduct a limited evaluation (in Section 5.2.6) with an autoregressive workload, GPT2-xl (1.5B parameters) [31], to demonstrate KACE's applicability to modern DL workloads.
Comparison baselines. We compare KACE with several baselines, including impractical ones, to evaluate its performance.
• Oracle is an impractical, oine-optimal policy that runs all possible colocation combinations for a given target workload and selects the pair that achieves the highest throughput sum. • Xu et al. [42] developed a method to predict workload interference among co-located VMs using GPU and CPU system metrics to train a Random Forest model. Their approach involves classifying GPU kernels as long or short to estimate context switch costs, and can be applied to non-virtualized environments as well. We implement their policy by collecting the metrics mentioned in their work and training a Random Forest model to predict the throughput sum of each colocation pair. • Random reports the mean of throughput sums of all possible colocations. For example, a given target workload could potentially be colocated with any of the 21 workloads we experiment with. Random obtains the throughput sum for all 21 colocations (except those that do not t in memory) and reports the mean. • MEMCAP, SM%, and MEM% are rule-based selection policies for choosing the workload to colocate with the target workload. Specically, MEMCAP selects, for colocation with the target workload, the workload that has the smallest GPU memory footprint metric. SM% and MEM% select, for colocation, the workload that has the highest Streaming Multiprocessor (SM) utilization or the lowest GPU memory bandwidth utilization, respectively. All three metrics can be easily obtained using nvidia-smi [25]. • Best rule based picks the best-performing rule from among MEMCAP, SM%, and MEM%, for each colocation scenario. This can be considered an oine policy as it requires running all rule-based policies and selecting the best.
This section presents our key experimental results. We start by discussing our performance prediction results under different ML models and training set sizes. Then we present our workload colocation results, highlighting the throughput sum gain aorded by KACE compared to various baselines.
We evaluate the accuracy of KACE for predicting throughput sum of colocated pairs of workloads using four model candidates: Random Forest (RF), a 3-layer 128⇥64⇥32 Deep Neural Network (NN ), H2O Automatic Machine Learning (AutoML) [13], and Linear Regression (LR). For RF, we perform a grid search, testing 100 congurations out of a search space of 4,320 to nd the optimal hyperparameters. For NN, we apply early stopping to prevent overtting. AutoML is an automated ML framework that selects optimal models from Distributed RF, Gradient Boosting, DL, and ensemble models within a given time constraint. We allocate 1 minute for AutoML to nd the optimal model. We report prediction results on the test set for dierent training data set sizes.
The top two graphs in Figure 2 show the R 2 and RMSE values, respectively, for dierent ML techniques as a function of the training data set size (reported as a % of the total data set size). We see that the dierent ML techniques evaluated result in slightly dierent R 2 and RMSE values, with AutoML performing the best among them and Random Forst (RF) performing the worst. The superior performance of AutoML is to be expected as it sweeps through various ML techniques and employs the best one for each scenario. We nd that AutoML typically picks DNN model as the best technique. The poor performance of RF is likely due to the strong correlations in the feature set, leading to a lack of diversity in decision trees and resulting in inconsistency [37]. Additionally, given the relatively small training set that is common in experimental works like ours (due to the time and eort required to generate samples), RF may result in overtting [3].
The bottom graph of Figure 2 shows, on a log scale, the time it takes for each ML technique to be trained. Note that the training time does not change much with the training data set size as the number of training data samples is small (the total data set size is less than 200 samples). Of course, the eort required to generate more training data scales with the number of training samples. KACE_LR has the least training time, by far, among the techniques evaluated; this is to be expected as LR is a relatively simple technique. KACE_RF has the highest training time as we also perform a grid search here to look for the best hyperparameters, and RF is a more involved learning technique. KACE_AutoML also has to search through various models, and so requires time.
For the colocated workload prediction problem, a simple ML technique like LR performs adequately, even outperforming more complex techniques like RF (in terms of R 2 and RMSE values), for several reasons. First, features like SM throughput may have a strong linear relationship with the throughput sum, making LR an ideal t. Second, LR can extrapolate predictions beyond the range of training data based on regression modeling. Third, models such as LR, that have high explainability, can benet from valuable features, such as kernel metrics, that inuence the response variable (throughput sum, in our case).
We now present our workload colocation results for KACE. Figure 4 shows the normalized throughput sum achieved by KACE (using LR) and other baseline policies, averaged over all 21 test set workloads. The normalization for each test case is with respect to Oracle; as such, Oracle shows a value of 1. Starting with Random, we see that it only achieves 54% of the throughput sum achieved by the oine-optimal Oracle. This is not surprising as Random eectively colocates a given target workload with a random workload, which can result in signicant GPU contention and performance degradation for the colocated workloads.
The rule-based policies, MEMCAP, SM%, and MEM%, result in a wide range of results, with SM% performing even worse than Random, and MEMCAP and MEM% achieving 70%-80% of the performance gains aorded by Oracle. Recall that MEMCAP selects the workload for colocation that has the smallest GPU memory footprint. However, this metric provides limited information and cannot capture, for example, the model architecture details. SM% selects the workload for colocation that has the highest SM utilization; while this should maximize GPU utilization under colocation, it can lead to high resource contention. MEM% selects the colocation workload that has the smallest memory busy rate, favoring workloads with fewer data transfers and smaller input data. As such, this policy routinely overlooks colocation candidates such as speech recognition inference that perform frequent data transfers.
The Best rule-based policy selects the best of the three rules (MEMCAP, SM%, and MEM%) for each test set target workload, and as such has a higher throughput sum than the individual rule-based policies. While the Best rule-based policy is impractical, it serves as a good baseline policy. For example, we note that Best rule-based improves upon MEM-CAP by achieving a roughly 7% higher average throughout sum of 83.7%. This suggests that MEMCAP likely outperforms the other rule-based policies (SM% and MEM%) in individual test cases; on further inspection, we nd that MEMCAP is the best single rule-based policy in 13 out of the 21 test scenarios.
Xu et al. [42] achieves 81% of the throughput sum achieved by Oracle. This is slightly higher than that achieved by MEM-CAP, but lower than that achieved by the (impractical) Best rule-based policy. Given that the feature set of Xu et al. is more extensive than the singular feature employed by MEM-CAP, the above results suggest that the additional features in Xu et al. do not provide signicant benets. This highlights the importance of selecting useful features when predicting colocated performance under GPU sharing.
KACE outperforms all other comparison baselines, achieving close to 91% of the throughput sum achieved by Oracle. Compared to the next-best practical policy (since Best rule-based requires oine runs), Xu et al., KACE achieves an 11% higher average throughput sum (relatively speaking). The superior performance of KACE can be attributed to the kernel information that it employs as part of its feature set, which allows it to better estimate GPU resource contention. We note that Xu et al. also employ some kernel metrics, such as long/short kernel information, but these features aim to capture GPU context switches among VMs, which is dierent from our objective of maximizing throughput. 5.2.3 Results for individual workloads. To better understand the performance of KACE on individual test workloads, we plot the normalized throughput sum achieved by KACE, Xu et al., and MEMCAP for all 21 test set workload colocations in Figure 5; error bars represent the standard deviation over the 5 runs for each colocation. The '_bx' notation in the workload name refers to the batch size of x and the '-train'/'-inf' refers to the workload type. We do not show other policies as they are either impractical or perform worse, and to aid comparisons. Of the 21 colocations, we nd that KACE is the best policy in 13 cases. In these 13 cases, KACE outperforms Xu et al. and MEMCAP by 20% and 24%, respectively, on average, and by as much as 88% and 52%, respectively. Of the remaining 8 cases, Xu et al. is the best policy in 6 of them, and outperforms KACE by 7%, on average, and by as much as 14%. In 2 cases, MEMCAP is the best policy, and outperforms KACE by 6%, on average, and by as much as 7%.
We see that policy performance varies with dierent workloads. For example, both Xu et al. and MEMCAP underperform in the case of ViT inference, especially for batch size 2 ('ViT_b2-inf'). On further inspection, we nd that Xu et al. consistently selects ViT_b2-inf as the colocation candidate due to its low SM and MEM usage, underestimating the throughput benets of larger batch sizes, and thus fails to choose the optimal ViT_inf-b16 colocation candidate. In contrast, KACE additionally considers individual workload throughput, thereby often selecting the optimal colocation candidate. On the other hand, MEMCAP frequently chooses BERT_b2-inf as the colocation candidate due to its minimal memory footprint, but its high compute needs (92% SM busy rate) lead to signicant interference when paired with ViT_b2-inf, which has moderate compute needs (29% SM busy rate). KACE accounts for thread and register usage, incorporating GPU compute information, and thus outperforms MEMCAP. A similar behavior is also observed when colocating with the Whisper inference workload. 5.2.4 Results for unseen workloads. We now consider the challenging case where the target workload to be executed is not part of the training set (under any batch size); this could arise in practice either because a new workload is encountered at runtime or because the training set size is limited. We evaluated all 7 distinct workloads with the highest batch size, ensuring that the target workload, for any batch size, is not included in training. We nd that KACE continues to perform well, achieving ⇠92% of the throughput sum achieved by Oracle. Compared to Xu et al. and rule-based policies, KACE achieves 11% and 16% higher average throughput sum, respectively. The above results show that KACE performs well even for unseen workloads. 5.2.5 Results for multiple unseen workloads. We now demonstrate KACE's ability to handle multiple (more than 2) colocated workloads; we consider the unseen workload case. Specically, we evaluate two colocation congurations:
(1) a target, unseen workload to be colocated with two other candidate workloads; and (2) a target, unseen workload to be colocated with three other candidate workloads. Our experimental results (not included here) show that KACE continues to perform well, achieving 96% and 91% of the throughput sum compared to Oracle when colocating with two and three workloads, respectively. While KACE can be extended beyond four colocated workloads, as the size of deep learning models continues to increase, we do not expect that numerous workloads will be needed to saturate a GPU while maintaining performance eciency [4]. 5.2.6 Results for an unseen autoregressive workload. We now consider colocation with an autoregressive workload, given that such modern models can be dierent from traditional vision and NLP models. Specically, we experiment with the GPT2-xl model [31] with output token lengths of 10, 20, and 214 (the average token length in LMSYS-CHAT-1M dataset [22]). To maintain low proling overhead, we prole kernel metrics during the prell stage and the decoding stage of the rst token (representative of compute-light sequential token generation stages); the MEMCAP system metric captures the memory footprint of the entire decode stage as the MEMCAP value increases with output token length. We use the same training dataset as in Table 1 (that is, without including GPT2-xl) and test with GPT2-xl as the target, unseen workload to be colocated with another workload. We nd that KACE achieves 92.3% of the throughput sum compared to Oracle and is ⇠10% better than the next-best policy (Best rule-based). Xu et al. performs poorly in this case (57.8% of Oracle) as it does not consider the total memory footprint, an important feature for autoregressive models given their large output token length. The above results suggest that KACE can be extended to modern GPU workloads as well.
Performance prediction under colocation. Colocation performance predictions can be made using either online proling in the presence of interference or oine proling based on monitored metrics. While online proling oers high accuracy, it cannot proactively schedule workloads for colocation, which is the focus of our work.
Xu et al. [42] predict interference between colocated DL jobs that are run on VMs on the same GPU. The authors also focus on predicting the VM context switch costs, and so select features for training that reect these costs. As discussed in Section 5.2, KACE outperforms Xu et al. due to its more appropriate feature set. MISO [18] predicts the optimal MIG partitions by evaluating colocation speedup with MPS, requiring multiple online proles for each colocation to determine the best conguration. Horus [43] uses DL computation graphs during oine proling to predict GPU utilization when colocating DL training workloads. However, its objective of maximizing GPU utilization need not translate to maximizing throughput sum, as acknowledged by the authors of Horus. Further, as shown using the SM% approach in Figure 4, maximizing SM utilization can lead to poor throughput gains due to high interference. Finally, both Horus and MISO are primarily suited for training workloads due to checkpoints and oine runs, which are impractical for inference jobs with performance requirements.
We note that there are works that focus on performance prediction under colocation [7,15,23,36], but for non-GPU workloads. As such, these works cannot predict the colocated performance of GPU-based DL workloads as they are oblivious to GPU features. GPU scheduling for DL workloads. Prior GPU scheduling works involving temporal sharing aim to minimize the cost of state swapping between DL jobs. Gandiva [40] introduces a checkpoint-restart mechanism to transfer training states between the GPU and host memory. Salus [44] optimizes GPU state management to reduce context switches. There are also other works that focus on inference serving with temporal sharing [33,34]. KACE does not focus on temporal sharing of the GPU, and instead considers spatial sharing.
For spatial sharing of the GPU, Wavelet [39] and Zico [21] colocate forward and backward passes of training tasks to reduce peak memory usage. IADeep [4] predicts trainingspecic metrics when colocating and exploits long-running training jobs to stop/restart them for optimization. However, these solutions are specic to training workloads. GSLICE [9] and gpulet [5] focus on performance isolation for colocated GPU workloads with the objective being predictable performance, which is dierent from KACE's objective of maximizing throughput. SHEPHERD [45] and AlpaServe [20] improve GPU eciency for model serving, but do not primarily focus on interference-aware GPU colocation. Orion [35] reduces GPU interference via kernel-level scheduling through the colocation of memory-and compute-intensive kernels. However, Orion requires oine kernel labeling and specic CUDA support for eective scheduling. In contrast, KACE focuses on workload-level solutions, tackling GPU interference from the application perspective. Further, Orion aims to minimize interference among a given colocated workload set whereas KACE aims to nd workloads to colocate to maximize throughput. As such, KACE and Orion are somewhat complementary. Similarly, KACE can be considered complementary to (the AMD-specic) REEF [14].
This paper presents KACE, a framework for ecient GPU spatial sharing that accurately and quickly predicts interference among colocated DL workloads. Our key contributions include: (i) selecting a set of kernel and system metrics that provide valuable information to predict colocated performance; (ii) using limited and one-time oine and exclusive proling of individual workloads, eliminating the need for costly online proling; (iii) identifying a simple ML model that provides adequate colocation performance prediction accuracy with minimal training time and a small training dataset; and (iv) experimentally evaluating KACE over multiple training and inference workloads and against various baselines. Evaluation results show that KACE achieves over 90% of the throughput sum achieved by Oracle using only a fraction of the training data.