More Like this

1. MFNetSim: A Multi-Fidelity Network Simulation Framework for Multi-Traffic Modeling of Dragonfly Systems

   https://doi.org/10.1145/3729424  

   Wang, Xin; Brown, Kevin A; Ross, Robert B; Carothers, Christopher D; Lan, Zhiling (September 2025, ACM Transactions on Modeling and Computer Simulation)

   In high-performance computing (HPC), modern supercomputers typically provide exclusive computing resources to user applications. Nevertheless, the interconnect network is a shared resource for both inter-node communication and across-node I/O access, among co-running workloads, leading to inevitable network interference. In this study, we develop MFNetSim, a multi-fidelity modeling framework that enables simulation of multi-traffic simultaneously over the interconnect network, including inter-process communication and I/O traffic. By combining different levels of abstraction, MFNetSim can efficiently co-model the communication and I/O traffic occurring on HPC systems equipped with flash-based storage. We conduct simulation studies of hybrid workloads composed of traditional HPC applications and emerging ML applications on a 1,056-node Dragonfly system with various configurations. Our analysis provides various observations regarding how network interference affects communication and I/O traffic. 

   more » « less
2. ADAPT: An Event-Based Adaptive Collective Communication Framework

   https://doi.org/10.1145/3208040.3208054  

   Luo, Xi; Wu, Wei; Bosilca, George; Patinyasakdikul, Thananon; Wang, Linnan; Dongarra, Jack (June 2018, The 27th International Symposium on High-Performance Parallel and Distributed Computing)

   The increase in scale and heterogeneity of high-performance computing (HPC) systems predispose the performance of Message Passing Interface (MPI) collective communications to be susceptible to noise, and to adapt to a complex mix of hardware capabilities. The designs of state of the art MPI collectives heavily rely on synchronizations; these designs magnify noise across the participating processes, resulting in significant performance slowdown. Therefore, such design philosophy must be reconsidered to efficiently and robustly run on the large-scale heterogeneous platforms. In this paper, we present ADAPT, a new collective communication framework in Open MPI, using event-driven techniques to morph collective algorithms to heterogeneous environments. The core concept of ADAPT is to relax synchronizations, while mamtaining the minimal data dependencies of MPI collectives. To fully exploit the different bandwidths of data movement lanes in heterogeneous systems, we extend the ADAPT collective framework with a topology-aware communication tree. This removes the boundaries of different hardware topologies while maximizing the speed of data movements. We evaluate our framework with two popular collective operations: broadcast and reduce on both CPU and GPU clusters. Our results demonstrate drastic performance improvements and a strong resistance against noise compared to other state of the art MPI libraries. In particular, we demonstrate at least 1.3X and 1.5X speedup for CPU data and 2X and 10X speedup for GPU data using ADAPT event-based broadcast and reduce operations. 

   more » « less
3. TACCL: Guiding Collective Algorithm Synthesis using Communication Sketches

   Shah, Aashaka; Chidambaram, Vijay; Cowan, Meghan; Maleki, Saeed; Musuvathi, Madan; Mytkowicz, Todd; Nelson, Jacob; Saarikivi, Olli; Singh, Rachee (April 2023, USENIX)

   Machine learning models are increasingly being trained across multiple GPUs and servers. In this setting, data is transferred between GPUs using communication collectives such as ALLTOALL and ALLREDUCE, which can become a significant bottleneck in training large models. Thus, it is important to use efficient algorithms for collective communication. We develop TACCL, a tool that enables algorithm designers to guide a synthesizer into automatically generating algorithms for a given hardware configuration and communication collective. TACCL uses a novel communication sketch abstraction to get crucial information from the designer to significantly reduce the search space and guide the synthesizer towards better algorithms. TACCL also uses a novel encoding of the problem that allows it to scale beyond single-node topologies. We use TACCL to synthesize algorithms for three collectives and two hardware topologies: DGX-2 and NDv2. We demonstrate that the algorithms synthesized by TACCL outperform the Nvidia Collective Communication Library (NCCL) by up to 6.7x. We also show that TACCL can speed up end-to-end training of Transformer-XL and BERT models by 11%–2.3x for different batch sizes. 

   more » « less
4. MPI-Vector-IO: Parallel I/O and Partitioning for Geospatial Vector Data

   https://doi.org/10.1145/3225058.3225105  

   Puri, Satish; Paudel, Anmol; Prasad, Sushil K. (August 2018, Proceedings of the 47th International Conference on Parallel Processing (ICPP 2018).)

   In recent times, geospatial datasets are growing in terms of size, complexity and heterogeneity. High performance systems are needed to analyze such data to produce actionable insights in an efficient manner. For polygonal a.k.a vector datasets, operations such as I/O, data partitioning, communication, and load balancing becomes challenging in a cluster environment. In this work, we present MPI-Vector-IO, a parallel I/O library that we have designed using MPI-IO specifically for partitioning and reading irregular vector data formats such as Well Known Text. It makes MPI aware of spatial data, spatial primitives and provides support for spatial data types embedded within collective computation and communication using MPI message-passing library. These abstractions along with parallel I/O support are useful for parallel Geographic Information System (GIS) application development on HPC platforms. Performance evaluation is done on Lustre and GPFS filesystems. MPI-Vector-IO scales well with MPI processes and file size and achieves bandwidth up to 22 GB/s for common spatial data access patterns. We observed that independent file read functions performed better than collective functions in MPI-IO for contiguous access pattern on Lustre. In general, the I/O is improved by one to two orders of magnitude over real-world datasets using up to 1152 CPU cores. Spatial Join query is used as an exemplar to demonstrate an end-to-end application using MPI-Vector-IO. 

   more » « less
5. How much memory do you actually need? Insights from 2025 Multiple HPC Resources Memory Usage Analysis

   Simakov, Nikolay; Saeli, Connor; Jones, Matthhew D; Ruhela, Amit (April 2026, www.researchgate.net)

   With recent unprecedented price spikes for computer memory, high-performance computing (HPC) centers face a challenge to properly balance the computational and memory resources to provide sufficient capabilities for their users. We analyze HPC memory trends using two complementary data sources: historical Top500 records and production job memory usage from five heterogeneous HPC resources at four HPC centers. Using the Top500 list, we examine long-term trends in memory-related system characteristics and generate a five-year forecast. Using Slurm accounting data, we analyze observed job memory usage and classify jobs by execution pattern. We then develop a simple model to project memory requirements for future CPU compute nodes with different cores-per-node counts. The results show that memory per core has grown only slowly, whereas memory per socket has increased more rapidly because of rising cores-per-socket counts. Observed CPU job memory usage is often moderate, indicating that most workloads do not require uniformly large memory on all nodes. For a projected 192-core CPU node, a multi-tier configuration combining 256~GiB, 512~GiB, and 2~TiB nodes would support most of the 2025 CPU workload while preserving some headroom for future growth. Overall, the results support workload-informed, multi-tier memory provisioning as a practical response to rising memory costs in HPC system design. 

   more » « less