An official website of the United States government
Despite advancements in the areas of parallel and distributed computing, the complexity of programming on High Performance Computing (HPC) resources has deterred many domain experts, especially in the areas of machine learning and artificial intelligence (AI), from utilizing performance benefits of such systems. Researchers and scientists favor high-productivity languages to avoid the inconvenience of programming in low-level languages and costs of acquiring the necessary skills required for programming at this level. In recent years, Python, with the support of linear algebra libraries like NumPy, has gained popularity despite facing limitations which prevent this code from distributed runs. Here we present a solution which maintains both high level programming abstractions as well as parallel and distributed efficiency. Phylanx, is an asynchronous array processing toolkit which transforms Python and NumPy operations into code which can be executed in parallel on HPC resources by mapping Python and NumPy functions and variables into a dependency tree executed by HPX, a general purpose, parallel, task-based runtime system written in C++. Phylanx additionally provides introspection and visualization capabilities for debugging and performance analysis. We have tested the foundations of our approach by comparing our implementation of widely used machine learning algorithms to accepted NumPy standards.
more »
« less
SHMEM-ML: Leveraging OpenSHMEM and Apache Arrow for Scalable, Composable Machine Learning
SHMEM-ML is a domain specific library for distributed array computations and machine learning model training & inference. Like other projects at the intersection of machine learning and HPC (e.g. dask, Arkouda, Legate Numpy), SHMEM-ML aims to leverage the performance of the HPC software stack to accelerate machine learning workflows. However, it differs in a number of ways. First, SHMEM-ML targets the full machine learning workflow, not just model training. It supports a general purpose nd-array abstraction commonly used in Python machine learning applications, and efficiently distributes transformation and manipulation of this ndarray across the full system. Second, SHMEM-ML uses OpenSHMEM as its underlying communication layer, enabling high performance networking across hundreds or thousands of distributed processes. While most past work in high performance machine learning has leveraged HPC message passing communication models as a way to efficiently exchange model gradient updates, SHMEM-ML’s focus on the full machine learning lifecycle means that a more flexible and adaptable communication model is needed to support both fine and coarse grain communication. Third, SHMEM-ML works to interoperate with the broader Python machine learning software ecosystem. While some frameworks aim to rebuild that ecosystem from scratch on top of the HPC software stack, SHMEM-ML is built on top of Apache Arrow, an in-memory standard for data formatting and data exchange between libraries. This enables SHMEM-ML to share data with other libraries without creating copies of data. This paper describes the design, implementation, and evaluation of SHMEM-ML – demonstrating a general purpose system for data transformation and manipulation while achieving up to a 38× speedup in distributed training performance relative to the industry standard Horovod framework without a regression in model metrics.
more »
« less
- Award ID(s):
- 2029004
- PAR ID:
- 10432119
- Editor(s):
- Poole, Steve; Hernandez, Oscar; Baker, Matthew; Curtis, Tony
- Date Published:
- Journal Name:
- Lecture notes in computer science
- Volume:
- 13159
- ISSN:
- 1611-3349
- Page Range / eLocation ID:
- 111–125
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experi- ments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask offers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.more » « less
-
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experiments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask o"ers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.more » « less
-
This paper describes a generalizable framework for creating context-aware wall-time prediction models for HPC applications. This framework: (a) cost-effectively generates comprehensive application-specific training data, (b) provides an application-independent machine learning pipeline that trains different regression models over the training datasets, and (c) establishes context-aware selection criteria for model selection. We explain how most of the training data can be generated on commodity or contention-free cyberinfrastructure and how the predictive models can be scaled to the production environment with the help of a limited number of resource-intensive generated runs (we show almost seven-fold cost reductions along with better performance). Our machine learning pipeline does feature transformation, and dimensionality reduction, then reduces sampling bias induced by data imbalance. Our context-aware model selection algorithm chooses the most appropriate regression model for a given target application that reduces the number of underpredictions while minimizing overestimation errors. Index Terms—AI4CI, Data Science Workflow, Custom ML Models, HPC, Data Generation, Scheduling, Resource Estimationsmore » « less
-
null (Ed.)Data engineering is becoming an increasingly important part of scientific discoveries with the adoption of deep learning and machine learning. Data engineering deals with a variety of data formats, storage, data extraction, transformation, and data movements. One goal of data engineering is to transform data from original data to vector/matrix/tensor formats accepted by deep learning and machine learning applications. There are many structures such as tables, graphs, and trees to represent data in these data engineering phases. Among them, tables are a versatile and commonly used format to load and process data. In this paper, we present a distributed Python API based on table abstraction for representing and processing data. Unlike existing state-of-the-art data engineering tools written purely in Python, our solution adopts high performance compute kernels in C++, with an in-memory table representation with Cython-based Python bindings. In the core system, we use MPI for distributed memory computations with a data-parallel approach for processing large datasets in HPC clusters.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Journal Article:
- https://doi.org/10.1007/978-3-031-04888-3_7
