Despite extensive investigation of job scheduling in data-intensive
computation frameworks, less consideration has been given to
optimizing job partitioning for resource utilization and efficient
processing. Instead, partitioning and job sizing are a form of dark
art, typically left to developer intuition and trial-and-error style
experimentation.
In this work, we propose that just as job scheduling and resource
allocation are out-sourced to a trusted mechanism external to the
workload, so too should be the responsibility for partitioning data
as a determinant for task size. Job partitioning essentially involves
determining the partition sizes to match the resource allocation at
the finest granularity. This is a complex, multi-dimensional problem
that is highly application specific: resource allocation, computational
runtime, shuffle and reduce communication requirements,
and task startup overheads all have strong influence on the most effective
task size for efficient processing. Depending on the partition
size, the job completion time can differ by as much as 10 times!
Fortunately, we observe a general trend underlying the tradeoff
between full resource utilization and system overhead across different
settings. The optimal job partition size balances these two
conflicting forces. Given this trend, we design Libra to automate
job partitioning as a framework extension. We integrate Libra with
Spark and evaluate its performance on EC2. Compared to state-of-the-art techniques, Libra can reduce the individual job execution
time by 25% to 70%.
more »
« less
The Effect of System Utilization on Application Performance Variability
Application performance variability caused by network contention
is a major issue on dragonfly based systems. This work-in-progress
study makes two contributions. First, we analyze real workload
logs and conduct application experiments on the production system
Theta at Argonne to evaluate application performance variability.
We find a strong correlation between system utilization and performance
variability where a high system utilization (e.g., above 95%)
can cause up to 21% degradation in application performance. Next,
driven by this key finding, we investigate a scheduling policy to mitigate
workload interference by leveraging the fact that production
systems often exhibit diurnal utilization behavior and not all users
are in a hurry for job completion. Preliminary results show that
this scheduling design is capable of improving system productivity
(measured by scheduling makespan) as well as improving user-level
scheduling metrics such as user wait time and job slowdown.
more »
« less
- NSF-PAR ID:
- 10097525
- Date Published:
- Journal Name:
- ACM Digital Library
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
High performance computing (HPC) is undergoing significant changes. The emerging HPC applications comprise both compute- and data-intensive applications. To meet the intense I/O demand from emerging data-intensive applications, burst buffers are deployed in production systems. Existing HPC schedulers are mainly CPU-centric. The extreme heterogeneity of hardware devices, combined with workload changes, forces the schedulers to consider multiple resources (e.g., burst buffers) beyond CPUs, in decision making. In this study, we present a multi-resource scheduling scheme named BBSched that schedules user jobs based on not only their CPU requirements, but also other schedulable resources such as burst buffer. BBSched formulates the scheduling problem into a multi-objective optimization (MOO) problem and rapidly solves the problem using a multi-objective genetic algorithm. The multiple solutions generated by BBSched enables system managers to explore potential tradeoffs among various resources, and therefore obtains better utilization of all the resources. The trace-driven simulations with real system workloads demonstrate that BBSched improves scheduling performance by up to 41% compared to existing methods, indicating that explicitly optimizing multiple resources beyond CPUs is essential for HPC scheduling.more » « less
-
Dragonfly class of networks are considered as promising interconnects for next-generation supercomputers. While Dragonfly+ networks offer more path diversity than the original Dragonfly design, they are still prone to performance variability due to their hierarchical architecture and resource sharing design. Event-driven network simulators are indispensable tools for navigating complex system design. In this study, we quantitatively evaluate a variety of application communication interactions on a 3,456-node Dragonfly+ system by using the CODES toolkit. This study looks at the impact of communication interference from a user’s perspective. Specifically, for a given application submitted by a user, we examine how this application will behave with the existing workload running in the system under different job placement policies. Our simulation study considers hundreds of experiment configurations including four target applications with representative communication patterns under a variety of network traffic conditions. Our study shows that intra-job interference can cause severe performance degradation for communication-intensive applications. Inter-job interference can generally be reduced for applications with one-toone or one-to-many communication patterns through job isolation. Application with one-to-all communication pattern is resilient to network interference.more » « less
-
Compute heterogeneity is increasingly gaining prominence in modern datacenters due to the addition of accelerators like GPUs and FPGAs. We observe that datacenter schedulers are agnostic of these emerging accelerators, especially their resource utilization footprints, and thus, not well equipped to dynamically provision them based on the application needs. We observe that the state-of-the-art datacenter schedulers fail to provide fine-grained resource guarantees for latency-sensitive tasks that are GPU-bound. Specifically for GPUs, this results in resource fragmentation and interference leading to poor utilization of allocated GPU resources. Furthermore, GPUs exhibit highly linear energy efficiency with respect to utilization and hence proactive management of these resources is essential to keep the operational costs low while ensuring the end-to-end Quality of Service (QoS) in case of user-facing queries.Towards addressing the GPU orchestration problem, we build Knots, a GPU-aware resource orchestration layer and integrate it with the Kubernetes container orchestrator to build Kube- Knots. Kube-Knots can dynamically harvest spare compute cycles through dynamic container orchestration enabling co-location of latency-critical and batch workloads together while improving the overall resource utilization. We design and evaluate two GPU-based scheduling techniques to schedule datacenter-scale workloads through Kube-Knots on a ten node GPU cluster. Our proposed Correlation Based Prediction (CBP) and Peak Prediction (PP) schemes together improves both average and 99 th percentile cluster-wide GPU utilization by up to 80% in case of HPC workloads. In addition, CBP+PP improves the average job completion times (JCT) of deep learning workloads by up to 36% when compared to state-of-the-art schedulers. This leads to 33% cluster-wide energy savings on an average for three different workloads compared to state-of-the-art GPU-agnostic schedulers. Further, the proposed PP scheduler guarantees the end-to-end QoS for latency-critical queries by reducing QoS violations by up to 53% when compared to state-of-the-art GPU schedulers.more » « less
-
null (Ed.)Modern high-performance computing (HPC) systems concurrently execute multiple distributed applications that contend for the high-speed network leading to congestion. Consequently, application runtime variability and suboptimal system utilization are observed in production systems. To address these problems, we propose Netscope, a congestion mitigation framework based on a novel delay sensitivity metric. Delay sensitivity of an application is used to quantify the impact of congestion on its runtime. Netscope uses delay sensitivity estimates to drive a congestion mitigation mechanism to selectively throttle applications that are less susceptible to congestion. We evaluate Netscope on two Cray Aries systems, including a production supercomputer, on common scientific applications. Our evaluation shows that Netscope has a low training cost and accurately estimates the impact of congestion on application runtime with a correlation between 0.7 and 0.9. Moreover, Netscope reduces application tail runtime increase by up to 16.3x while improving the median system utility by 12%.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- The DOI is not currently available.
