The popularity of Python in various application domains has driven widespread adoption of NumPy-style APIs. To improve performance, libraries such as PyTorch, JAX, CuPy, and cuPyNumeric offer GPU-compatible counterparts to NumPy functions. However, substituting NumPy with these alternatives is not always beneficial due to the overheads of type conversion, data transfer, and kernel launch costs. We present SmartDispatch, a runtime framework that dynamically substitutes NumPy-style API calls with semantically equivalent implementations from other libraries to improve performance. Our system includes a knowledge base of equivalent APIs, a hardware-aware microbenchmarking component to identify substitution thresholds, and a runtime substitution tool. Evaluation on four platforms with varying CPU-GPU architectures using machine learning models from real-world benchmarks shows that consistent performance gains (1.3× to 5.8×) can be achieved without requiring code modification, demonstrating the effectiveness of cross-library substitution in heterogeneous environments.
more »
« less
Offload Annotations: Bringing Heterogeneous Computing to Existing Libraries and Workloads
As specialized hardware accelerators such as GPUs become increasingly popular, developers are looking for ways to target these platforms with high-level APIs. One promising approach is kernel libraries such as PyTorch or cuML, which provide interfaces that mirror CPU-only counterparts such as NumPy or Scikit-Learn. Unfortunately, these libraries are hard to develop and to adopt incrementally: they only support a subset of their CPU equivalents, only work with datasets that fit in device memory, and require developers to reason about data placement and transfers manually. To address these shortcomings, we present a new approach called offload annotations (OAs) that enables heterogeneous GPU computing in existing workloads with few or no code modifications. An annotator annotates the types and functions in a CPU library with equivalent kernel library functions and provides an offloading API to specify how the inputs and outputs of the function can be partitioned into chunks that fit in device memory and transferred between devices. A runtime then maps existing CPU functions to equivalent GPU kernels and schedules execution, data transfers and paging. In data science workloads using CPU libraries such as NumPy and Pandas, OAs enable speedups of up to 1200⇥ and a median speedup of 6.3⇥ by transparently offloading functions to a GPU using existing kernel libraries. In many cases, OAs match the performance of handwritten heterogeneous implementations. Finally, OAs can automatically page data in these workloads to scale to datasets larger than GPU memory, which would need to be done manually with most current GPU libraries.
more »
« less
- Award ID(s):
- 1651570
- PAR ID:
- 10213413
- Date Published:
- Journal Name:
- USENIX ATC 2020
- Subject(s) / Keyword(s):
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Data movement between main memory and the CPU is a major bottleneck in parallel data-intensive applications. In response, researchers have proposed using compilers and intermediate representations (IRs) that apply optimizations such as loop fusion under existing high-level APIs such as NumPy and TensorFlow. Even though these techniques generally do not require changes to user applications, they require intrusive changes to the library itself: often, library developers must rewrite each function using a new IR. In this paper, we propose a new technique called split annotations (SAs) that enables key data movement optimizations over unmodified library functions. SAs only require developers to annotate functions and implement an API that specifies how to partition data in the library. The annotation and API describe how to enable cross-function data pipelining and parallelization, while respecting each function's correctness constraints. We implement a parallel runtime for SAs in a system called Mozart. We show that Mozart can accelerate workloads in libraries such as Intel MKL and Pandas by up to 15x, with no library modifications. Mozart also provides performance gains competitive with solutions that require rewriting libraries, and can sometimes outperform these systems by up to 2x by leveraging existing hand-optimized code.more » « less
-
An increasing number of applications benefit from heterogeneous hardware accelerators. Such accelerators often require the application to manually manage memory buffers on devices and transfer data between host and device buffers. A programming model that unifies the virtual address space across the host and devices is appealing because it enables automatic memory transfers and simplifies application-level programming. However, the automatic memory transfers can sometimes be redundant, which decreases performance. NVIDIA’s UVM (unified virtual memory) driver provides a unified virtual address space for CPU-GPU programming. This paper identifies redundant memory transfers (RMTs) as a common performance issue with UVM. To address this issue, this paper proposes a data discard directive, and evaluates two implementations of that directive, UvmDiscard and UvmDiscardLazy. This directive exploits application-level knowledge to avoid RMTs. The implementations were integrated with NVIDIA’s open-source UVM driver to demonstrate their usefulness on real-world CUDA UVM applications. For example, the use of the discard directive increases training throughput by 61.2% on a large deep learning application that oversubscribes GPU memory.more » « less
-
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experiments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask o"ers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.more » « less
-
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experi- ments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask offers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.more » « less
An official website of the United States government

