More Like this

1. Accelerating AWP-ODC for Large-scale Earthquake Simulations Using MVAPICH2

   Palla, A; Wang, S; Zhang, T; Cui, Y (August 2025, Annual MVAPICH User Group (MUG) Conference)

   Accurate simulation of earthquake scenarios is essential for advancing seismic hazard analysis and risk mitigation strategies. At the San Diego Supercomputer Center (SDSC), our research focuses on optimizing the performance and reliability of large-scale earthquake simulations using the AWP-ODC software. By implementing GPU-aware MPI calls, we enable direct data processing within GPU memory, eliminating the need for explicit data transfers between CPU and GPU. This GPU-aware MPI achieves nearly ideal parallel efficiency at full scale across both Nvidia and AMD GPUs, leveraging the MVAPICH-PLUS support on Frontier at Oak Ridge National Laboratory and Vista at the Texas Advanced Computing Center. We utilized the MVAPICH-Plus 4.0 compiler to enable ZFP compression, which significantly enhances inter-node communication efficiency – a critical improvement given the communication bottleneck inherent in large-scale simulations. Our GPU-aware AWP-ODC versions include linear forward, topography and nonlinear Iwan-type solvers with discontinuous mesh support. On the Frontier system with MVAPICH 4.0, Hip-aware MPI calls on MI250X GPUs deliver nearly ideal weak-scaling speedup up to 8,192 nodes for both linear and topography versions. On TACC’s Vista system, CUDA-aware MPI calls on GH200 GPUs substantially outperform their non-GPU-aware counterparts across all three solver versions. This poster will present a detailed evaluation of GPU-aware AWP-ODC using MVAPICH, including the impact of ZFP message compression compared to the native versions. Our results highlight the importance of Mvapich support for GPU-ware MPI and on-the-fly compression techniques for accelerating and scaling earthquake simulations. 

   more » « less
2. AWP-ODC with Topography and Discontinuous Mesh:Extreme-Scale Earthquake Simulation using MVAPICH

   Cui, Y; Yeh, T (August 2024, Annual MVAPICH USER Group (MUG) Meeting)

   The Gordon Bell-winning AWP-ODC application continues to push the boundaries of earthquake simulation by leveraging the enhanced performance of MVAPICH on both CPU and GPU based architectures. This presentation highlights the recent improvements to the code and its application to broadband deterministic 3D wave propagation simulations of earthquake ground motions, incorporating high-resolution surface topography and detailed underground structures. The results of these simulations provide critical insights into the potential impacts of major earthquakes, contributing to more effective disaster preparedness and mitigation strategies. Additionally, the presentation will address the scientific and technical challenges encountered during the process and discuss the implications for future large-scale seismic studies on Exascale computing systems. 

   more » « less
3. GPU-aware support to improve the performance of collective communication on GPUs

   Cui, Y; Palla, A; Zhang, T; Wang, S; Roten, D; Koesterke, L; Zhang, Z; Maechling, P (September 2025, SCEC Publications)

   We have implemented GPU-aware support across all AWP-ODC versions and enhanced message-passing collective communications for this memory-bound finite-difference solver. This provides cutting-edge communication support for production simulations on leadership-class computing facilities, including OLCF Frontier and TACC Vista. We achieved significant performance gains, reaching 37 sustained Petaflop/s and reducing time-to-solution by 17.2% using the GPU-aware feature on 8,192 Frontier nodes, or 65,336 MI250X GCDs. The AWP-ODC code has also been optimized for TACC Vista, an Arm-based NVIDIA GH200 Grace Hopper Superchip, demonstrating excellent application performance. This poster will showcase studies and GPU performance characteristics. We will discuss our verification of GPU-aware development and the use of high-performance MVAPICH libraries, including on-the-fly compression, on modern GPU clusters. 

   more » « less
4. Porting topography version of AWP-ODC to OLCF Frontier, with ROCm-Aware support to improve the performance of collective communication

   Cui, Y; Palla, A; Talreja, A; Koesterke, L; Zhang, W; Yeh, T; Maechling, P (September 2024, SCEC Publications)

   We have ported and verified the topography version of AWP-ODC, with discontinuous mesh feature enabled, to HIP so that it runs on AMD MI250X GPUs. 103.3% parallel efficiency was benchmarked on Frontier between 8 and 4,096 nodes or up to 32,768 GCDs. Frontier is a two exaflop/s computing system based on the AMD Radeon Instinct GPUs and EPYC CPUs, a Leadership Computing Facility at Oak Ridge National Laboratory (ORNL). This HIP topography code has been used in the production runs on Frontier, a primary computing engine currently utilizing the 2024 SCEC INCITE allocation, a 700K node-hours supercomputing time award. Furthermore, we implemented ROCm-Aware GPU direct support in the topo code, and demonstrated 14% additional reduction in time-to-solution up to 4,096 nodes. The AWP-ODC-Topo code is also tuned on TACC Vista, an Arm-based NVIDIA GH200 Grace Hopper Superchip, with excellent performance demonstrated. This poster will demonstrate the studies of weak scaling and the performance characteristics on GPUs. We discuss the efforts of verifying the ROCm-Aware development, and utilizing high-performance MVAPICH libraries with the on-the-fly compression on modern GPU clusters. 

   more » « less
5. Earthquake behaviors in earthquake cycle simulations with fault damage zones

   Huang, Yihe; Thakur, Prithvi (January 2022, SSA Annual Meeting)

   Faults are usually surrounded by damage zones associated with localized deformation. Here we use fully dynamic earthquake cycle simulations to quantify the behaviors of earthquakes in fault damage zones. We show that fault damage zones can make a significant contribution to the spatial and temporal seismicity distribution. Fault stress heterogeneities generated by fault zone waves persist over multiple earthquake cycles that, in turn, produce small earthquakes that are absent in homogeneous simulations with the same friction conditions. Shallow fault zones can produce a bimodal depth distribution of earthquakes with clustering of seismicity at both shallower and deeper depths. Fault zone healing during the interseismic period also promotes the penetration of aseismic slip into the locked region and reduces the sizes of fault asperities that host earthquakes. Hence, small and moderate subsurface earthquakes with irregular recurrence intervals are commonly observed in immature fault zone simulations with interseismic healing. To link our simulation results to geological observations, we will use simulated fault slip at different depths to infer the timing and recurrence intervals of earthquakes and discuss how such measurements can affect our understanding of earthquake behaviors. We will also show that the maturity and material properties of fault damage zones have strong influence on whether long-term earthquake characteristics are represented by single events. 

   more » « less