An official website of the United States government
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experi- ments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask offers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.
more »
« less
This content will become publicly available on May 3, 2027
A Comparative Evaluation of Imputation Models for Agricultural Weather Networks
High-resolution weather data are essential for irrigation scheduling, frost protection, and pest and disease risk modeling. However, weather stations frequently experience multi-hour to multi-day outages, leading to substantial downtime for weather-driven decision-making. To mitigate this, stakeholders often rely on “imputation” models to reconstruct missing data. Despite the existence of many statistical and machine-learning models, it remains unclear which imputation methods are most suitable for operational agricultural settings. This paper evaluates twelve imputation methods—statistical models, classical machine learning algorithms, and deep neural networks—to identify the most suitable model for agricultural applications. We tested the models using data from five meteorological towers (three in Jena, Germany, and two in Sunnyside, Washington, USA). We perform a thorough performance-engineering study: in addition to accuracy, we evaluate runtime, inference throughput, peak memory usage, GPU usage, energy consumption, and monetary cost. Our findings are surprising: among all complex and advanced (neural network) models, a properly tuned Random Forest (RF) model consistently outperforms them across multiple evaluation categories (e.g., accuracy, latency, throughput, and cost). Specifically, an RF achieves competitive error with no GPU dependencies, modest memory usage, and substantially lower energy expenditure than deep learning baselines. Our research shows that classical machine learning models remain a compelling choice for scalable, cost-aware weather data imputation in agricultural decision-support systems.
more »
« less
- Award ID(s):
- 2345653
- PAR ID:
- 10683263
- Publisher / Repository:
- ACM
- Date Published:
- ISBN:
- 9798400723254
- Page Range / eLocation ID:
- 230 to 241
- Format(s):
- Medium: X
- Location:
- Florence Italy
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
This paper presents a comparative performance analysis of three popular Python data manipulation libraries—Pandas, Polars, and Dask—within the context of deep learning training pipelines. The existing studies in this area do not embed the libraries inside a full deep-learning training pipeline where data loading, preprocessing, and batch feeding interact tightly with GPU workloads. To bridge this gap, we integrate Pandas, Polars, and Dask into representative deep learning training and inference pipelines and conduct experiments across a wide range of various machine learning models and datasets, measuring key performance indicators such as runtime, memory usage, disk usage, and energy consumption (CPU and GPU). Our comprehensive analysis reveals that Polars consistently minimizes CPU energy consumption on larger workloads, while Pandas remains competitive for moderate sizes. Dask’s overhead can lead to higher energy usage on small to moderate datasets. All three libraries achieve similar runtimes for heavy GPU workloads (ResNet, Mask R-CNN). Polars and Pandas maintain lower CPU memory footprints than Dask, but Dask o"ers easier scalability if data truly exceeds available RAM. Polars shows marginal energy savings on the CPU during preprocessing.more » « less
-
Abstract Accurate prediction of crop yield is critical for supporting food security, agricultural planning, and economic decision-making. However, yield forecasting remains a significant challenge due to the complex and nonlinear relationships between weather variables and crop production, as well as spatial heterogeneity across agricultural regions. We propose a Spatially Varying Deep Functional Neural Network (SVD-funNet), a deep neural network architecture that integrates functional and scalar predictors with spatially varying coefficients and spatial random effects. The method is designed to flexibly model spatially indexed functional data, such as daily temperature curves, and their relationship to variability in the response, while accounting for spatial correlation. SVD-funNet mitigates the curse of dimensionality through a low-rank structure inspired by the spatially varying functional index model (SVFIM). Through comprehensive simulations, we demonstrate that SVD-funNet outperforms state-of-the-art functional regression models for spatial data, when the functional predictors exhibit complex structure and their relationship with the response varies spatially in a potentially nonstationary manner. Application to corn yield data from the U.S. Midwest demonstrates that SVD-funNet achieves superior predictive accuracy compared to both leading machine learning approaches and parametric statistical models. These results highlight the model’s robustness and its potential applicability to other weather-sensitive crops.more » « less
-
We study training of Graph Neural Networks (GNNs) for large-scale graphs. We revisit the premise of using distributed training for billion-scale graphs and show that for graphs that fit in main memory or the SSD of a single machine, out- of-core pipelined training with a single GPU can outperform state-of-the-art (SoTA) multi-GPU solutions. We introduce MariusGNN, the first system that utilizes the entire storage hierarchy—including disk—for GNN training. MariusGNN introduces a series of data organization and algorithmic contributions that 1) minimize the end-to-end time required for training and 2) ensure that models learned with disk-based training exhibit accuracy similar to those fully trained in memory. We evaluate MariusGNN against SoTA systems for learning GNN models and find that single-GPU training in MariusGNN achieves the same level of accuracy up to 8× faster than multi-GPU training in these systems, thus, introducing an order of magnitude monetary cost reduction. MariusGNN is open-sourced at www.marius-project.org.more » « less
-
Random Forests (RFs) are a commonly used machine learning method for classification and regression tasks spanning a variety of application domains, including bioinformatics, business analytics, and software optimization. While prior work has focused primarily on improving performance of the training of RFs, many applications, such as malware identification, cancer prediction, and banking fraud detection, require fast RF classification. In this work, we accelerate RF classification on GPU and FPGA. In order to provide efficient support for large datasets, we propose a hierarchical memory layout suitable to the GPU/FPGA memory hierarchy. We design three RF classification code variants based on that layout, and we investigate GPU- and FPGA-specific considerations for these kernels. Our experimental evaluation, performed on an Nvidia Xp GPU and on a Xilinx Alveo U250 FPGA accelerator card using publicly available datasets on the scale of millions of samples and tens of features, covers various aspects. First, we evaluate the performance benefits of our hierarchical data structure over the standard compressed sparse row (CSR) format. Second, we compare our GPU implementation with cuML, a machine learning library targeting Nvidia GPUs. Third, we explore the performance/accuracy tradeoff resulting from the use of different tree depths in the RF. Finally, we perform a comparative performance analysis of our GPU and FPGA implementations. Our evaluation shows that, while reporting the best performance on GPU, our code variants outperform the CSR baseline both on GPU and FPGA. For high accuracy targets, our GPU implementation yields a 5-9 × speedup over CSR, and up to a 2 × speedup over Nvidia’s cuML library.more » « less
