Search for: All records

ABSTRACT We have implemented and verified a parallel-series Iwan-type nonlinear model in a 3D fourth-order staggered-grid velocity–stress finite-difference method. The Masing unloading and reloading behavior is simulated by tracking an overlay of concentric von Mises yield surfaces. Lamé parameters and failure stresses pertaining to each surface are calibrated to reproduce the stress–strain backbone curve, which is controlled by the reference strain assigned to a given depth level. The implementation is successfully verified against established codes for 1D and 2D SH-wave benchmarks. The capabilities of the method for large-scale nonlinear earthquake modeling are demonstrated for an Mw 7.8 dynamic rupture ShakeOut scenario on the southern San Andreas fault. Although ShakeOut simulations with a single yield surface reduces long-period ground-motion amplitudes by about 25% inside a waveguide in greater Los Angeles, Iwan nonlinearity further reduces the values by a factor of 2. For example, inside the Whittier Narrows corridor spectral accelerations at a period of 3 s are reduced from 1g in the linear case to about 0.8 in the bilinear case and to 0.3–0.4g in the multisurface Iwan nonlinear case, depending on the choice of reference strain. Normalized shear modulus reductions reach values of up to 50% in the waveguide and up to 75% in the San Bernardino basin at the San Andreas fault. We expect the implementation to be a valuable tool for future nonlinear 3D dynamic rupture and ground-motion simulations in models with coupled source, path, and site effects. 

We have implemented GPU-aware support across all AWP-ODC versions and enhanced message-passing collective communications for this memory-bound finite-difference solver. This provides cutting-edge communication support for production simulations on leadership-class computing facilities, including OLCF Frontier and TACC Vista. We achieved significant performance gains, reaching 37 sustained Petaflop/s and reducing time-to-solution by 17.2% using the GPU-aware feature on 8,192 Frontier nodes, or 65,336 MI250X GCDs. The AWP-ODC code has also been optimized for TACC Vista, an Arm-based NVIDIA GH200 Grace Hopper Superchip, demonstrating excellent application performance. This poster will showcase studies and GPU performance characteristics. We will discuss our verification of GPU-aware development and the use of high-performance MVAPICH libraries, including on-the-fly compression, on modern GPU clusters. 

The SCEC CyberShake platform implements a repeatable scientific workflow to perform 3D physics-based probabilistic seismic hazard analysis (PSHA). Earlier this year we calculated CyberShake Study 24.8 for the San Francisco Bay Area. Study 24.8 includes both low-frequency and broadband PSHA models, calculated at 315 sites. This study required building a regional velocity model from existing 3D models, with a near-surface low-velocity taper and a minimum Vs of 400 m/s. Pegasus-WMS managed the execution of Study 24.8 for 45 days on the OLCF Frontier and TACC Frontera systems. 127 million seismograms and 34 billion intensity measures were produced and automatically transferred to SCEC storage. Study 24.8 used a HIP language implementation of the AWP-ODC wave propagation code on AMD-GPU Frontier nodes to produce strain Green tensors, which were convolved with event realizations to synthesize seismograms. Seismograms were processed to derive data products such as intensity measures, site-specific hazard curves and regional hazard maps. CyberShake combines 3D low-frequency deterministic (≤1 Hz) simulations with high-frequency calculations using stochastic modules from the Broadband Platform to produce results up to 25 Hz, with validation performed using historical events. New CyberShake data products from this study include vertical seismograms, vertical response spectra, and period-dependent significant durations. The presented results include comparisons of hazard estimates between Study 24.8, the previous CyberShake study for this region (18.8), and the NGA-West2 ground motion models (GMMs). We find that Study 24.8 shows overall lower hazard than 18.8, likely due to changes in rupture coherency, with the exception of a few regions: 24.8 shows higher hazard than both the GMMs and 18.8 at long periods in the Livermore area, due to deepening of the Livermore basin in the velocity model, as well as higher hazard east of San Pablo Bay and south of San Jose. At high frequencies, Study 24.8 hazard is lower than that of the GMMs, reflecting reduced variability in the stochastic components. We are also using CyberShake ground motion data to investigate the effects of preferred rupture directions on site-specific hazard. By default, PSHA hazard products assume all events on a given fault and magnitude are equally likely, but by varying these probabilities we can examine the effects of preferred rupture directions on given faults on CyberShake hazard estimates. 

Accurate simulation of earthquake scenarios is essential for advancing seismic hazard analysis and risk mitigation strategies. At the San Diego Supercomputer Center (SDSC), our research focuses on optimizing the performance and reliability of large-scale earthquake simulations using the AWP-ODC software. By implementing GPU-aware MPI calls, we enable direct data processing within GPU memory, eliminating the need for explicit data transfers between CPU and GPU. This GPU-aware MPI achieves nearly ideal parallel efficiency at full scale across both Nvidia and AMD GPUs, leveraging the MVAPICH-PLUS support on Frontier at Oak Ridge National Laboratory and Vista at the Texas Advanced Computing Center. We utilized the MVAPICH-Plus 4.0 compiler to enable ZFP compression, which significantly enhances inter-node communication efficiency – a critical improvement given the communication bottleneck inherent in large-scale simulations. Our GPU-aware AWP-ODC versions include linear forward, topography and nonlinear Iwan-type solvers with discontinuous mesh support. On the Frontier system with MVAPICH 4.0, Hip-aware MPI calls on MI250X GPUs deliver nearly ideal weak-scaling speedup up to 8,192 nodes for both linear and topography versions. On TACC’s Vista system, CUDA-aware MPI calls on GH200 GPUs substantially outperform their non-GPU-aware counterparts across all three solver versions. This poster will present a detailed evaluation of GPU-aware AWP-ODC using MVAPICH, including the impact of ZFP message compression compared to the native versions. Our results highlight the importance of Mvapich support for GPU-ware MPI and on-the-fly compression techniques for accelerating and scaling earthquake simulations. 

We integrate GPU-aware MVAPICH2 in AWP-ODC, a scalable finite difference code for wave propagation in nonlinear media. On OLCF Frontier, HIP-aware MVAPICH2 yields a 17.8% T2S improvement over the non-GPU-aware version and achieves 95% parallel efficiency on 65,536 AMD MI250X GCDs. On TACC Vista, CUDA-aware MVAPICH2 delivers a 3.5% performance gain across 2-256 Nvidia GH200 GPUs, with parallel efficiencies of 82% in the linear case and 92% in the computationally more intense nonlinear case. We deploy the code for production-scale earthquake simulations on leadership-class systems 

We have ported and verified the topography version of AWP-ODC, with discontinuous mesh feature enabled, to HIP so that it runs on AMD MI250X GPUs. 103.3% parallel efficiency was benchmarked on Frontier between 8 and 4,096 nodes or up to 32,768 GCDs. Frontier is a two exaflop/s computing system based on the AMD Radeon Instinct GPUs and EPYC CPUs, a Leadership Computing Facility at Oak Ridge National Laboratory (ORNL). This HIP topography code has been used in the production runs on Frontier, a primary computing engine currently utilizing the 2024 SCEC INCITE allocation, a 700K node-hours supercomputing time award. Furthermore, we implemented ROCm-Aware GPU direct support in the topo code, and demonstrated 14% additional reduction in time-to-solution up to 4,096 nodes. The AWP-ODC-Topo code is also tuned on TACC Vista, an Arm-based NVIDIA GH200 Grace Hopper Superchip, with excellent performance demonstrated. This poster will demonstrate the studies of weak scaling and the performance characteristics on GPUs. We discuss the efforts of verifying the ROCm-Aware development, and utilizing high-performance MVAPICH libraries with the on-the-fly compression on modern GPU clusters. 

The ShakeOut simulations of a M7.8 earthquake on the southern San Andreas fault (2008) studies predicted unexpectedly large ground motions throughout Southern California due to waveguide e#ects from interconnected sedimentary basins in 3D velocity model. Here, we re-examine the ground motion predictions from the ShakeOut scenario using the most recent updates on the velocity models and the realistic surface topography from the Digital Elevation Model. The exceptional scalability and performance of the AWP-ODC due to the most recent advancements allows for examination of plenty earth models including the irregular surface topography at low computational cost. 

The Gordon Bell-winning AWP-ODC application continues to push the boundaries of earthquake simulation by leveraging the enhanced performance of MVAPICH on both CPU and GPU based architectures. This presentation highlights the recent improvements to the code and its application to broadband deterministic 3D wave propagation simulations of earthquake ground motions, incorporating high-resolution surface topography and detailed underground structures. The results of these simulations provide critical insights into the potential impacts of major earthquakes, contributing to more effective disaster preparedness and mitigation strategies. Additionally, the presentation will address the scientific and technical challenges encountered during the process and discuss the implications for future large-scale seismic studies on Exascale computing systems. 

AWP-ODC is a 4th-order finite difference code used for linear wave propagation, Iwan-type nonlinear dynamic rupture and wave propagation, and Strain Green Tensor simulation2. We have ported and verified the linear and topography version of AWP-ODC, with discontinuous mesh as well as topography, to HIP so that it can also run on AMD GPUs. The topography code achieved a 99.6% parallel efficiency on 4,096 nodes on Frontier, a Leadership Computing Facility at Oak Ridge National Laboratory. We have also implemented CUDA-aware features and on-the-fly GDR compression in the linear version of the ported HIP code. These enhancements significantly improve data transfer efficiency between GPUs, reducing communication overhead and boosting overall performance. We have also extended CUDA-aware features to the topography version and are actively working on incorporating GDR compression into this version as well. We see 154% benefits over IMPI in MVAPICH2-GDR with CUDA-aware support and on-the-fly compression for linear AWP-ODC on Lonestar-6 A100 nodes. Furthermore, we have successfully integrated a checkpointing feature into the nonlinear IWAN version of AWP-ODC, prepared for future extreme-scale simulation during Texascale Days of Frontera at TACC.