skip to main content


Title: Extreme-scale Earthquake Simulation with MVAPICH
The Gordon Bell winning AWP-ODC application has a long history of boosted performance with MVAPICH on both CPU and GPU-based architectures. This talk will highlight a recent compression support implemented by the MVAPICH team, and its benefits to the large-scale earthquake simulation on the leadership class computing systems. The presentation will conclude with a discussion of the opportunities and technical challenges associated with the development of earthquake simulation software for Exascale computing.  more » « less
Award ID(s):
2311833
PAR ID:
10538034
Author(s) / Creator(s):
Publisher / Repository:
Annual MVAPICH USER Group (MUG) Meeting
Date Published:
Format(s):
Medium: X
Location:
Columbus
Sponsoring Org:
National Science Foundation
More Like this
  1. The Gordon Bell-winning AWP-ODC application continues to push the boundaries of earthquake simulation by leveraging the enhanced performance of MVAPICH on both CPU and GPU based architectures. This presentation highlights the recent improvements to the code and its application to broadband deterministic 3D wave propagation simulations of earthquake ground motions, incorporating high-resolution surface topography and detailed underground structures. The results of these simulations provide critical insights into the potential impacts of major earthquakes, contributing to more effective disaster preparedness and mitigation strategies. Additionally, the presentation will address the scientific and technical challenges encountered during the process and discuss the implications for future large-scale seismic studies on Exascale computing systems. 
    more » « less
  2. Faults are usually surrounded by damage zones associated with localized deformation. Here we use fully dynamic earthquake cycle simulations to quantify the behaviors of earthquakes in fault damage zones. We show that fault damage zones can make a significant contribution to the spatial and temporal seismicity distribution. Fault stress heterogeneities generated by fault zone waves persist over multiple earthquake cycles that, in turn, produce small earthquakes that are absent in homogeneous simulations with the same friction conditions. Shallow fault zones can produce a bimodal depth distribution of earthquakes with clustering of seismicity at both shallower and deeper depths. Fault zone healing during the interseismic period also promotes the penetration of aseismic slip into the locked region and reduces the sizes of fault asperities that host earthquakes. Hence, small and moderate subsurface earthquakes with irregular recurrence intervals are commonly observed in immature fault zone simulations with interseismic healing. To link our simulation results to geological observations, we will use simulated fault slip at different depths to infer the timing and recurrence intervals of earthquakes and discuss how such measurements can affect our understanding of earthquake behaviors. We will also show that the maturity and material properties of fault damage zones have strong influence on whether long-term earthquake characteristics are represented by single events. 
    more » « less
  3. null (Ed.)
    Since its commissioning in 2004, the UC San Diego Large High-Performance Outdoor Shake Table (LHPOST) has enabled the seismic testing of large structural, geostructural and soil-foundation-structural systems, with its ability to accurately reproduce far- and near-field ground motions. Thirty-four (34) landmark projects were conducted on the LHPOST as a national shared-use equipment facility part of the National Science Foundation (NSF) Network for Earthquake Engineering Simulation (NEES) and currently Natural Hazards Engineering Research Infrastructure (NHERI) programs, and an ISO/IEC Standard 17025:2005 accredited facility. The tallest structures ever tested on a shake table were conducted on the LHPOST, free from height restrictions. Experiments using the LHPOST generate essential knowledge that has greatly advanced seismic design practice and response predictive capabilities for structural, geostructural, and non-structural systems, leading to improved earthquake safety in the community overall. Indeed, the ability to test full-size structures has made it possible to physically validate the seismic performance of various systems that previously could only be studied at reduced scale or with computer models. However, the LHPOST's limitation of 1-DOF (uni-directional) input motion prevented the investigation of important aspects of the seismic response of 3-D structural systems. The LHPOST was originally conceived as a six degrees-of-freedom (6-DOF) shake table but built as a single degree-of-freedom (1-DOF) system due to budget limitations. The LHPOST is currently being upgraded to 6-DOF capabilities. The 6-DOF upgraded LHPOST (LHPOST6) will create a unique, large-scale, high-performance, experimental research facility that will enable research for the advancement of the science, technology, and practice in earthquake engineering. Testing of infrastructure at large scale under realistic multi-DOF seismic excitation is essential to fully understand the seismic response behavior of civil infrastructure systems. The upgraded 6-DOF capabilities will enable the development, calibration, and validation of predictive high-fidelity mathematical/computational models, and verifying effective methods for earthquake disaster mitigation and prevention. Research conducted using the LHPOST6 will improve design codes and construction standards and develop accurate decision-making tools necessary to build and maintain sustainable and disaster-resilient communities. Moreover, it will support the advancement of new and innovative materials, manufacturing methods, detailing, earthquake protective systems, seismic retrofit methods, and construction methods. This paper will provide a brief overview of the 1-DOF LHPOST and the impact of some past landmark projects. It will also describe the upgrade to 6-DOF and the new seismic research and testing that the LHPOST6 facility will enable. 
    more » « less
  4. Rodriguez, Julio A. (Ed.)

    Design code-based “life-safety” requirements for structural earthquake and tsunami design offer reasonable guidelines to construct buildings that will remain standing during a tsunami or seismic event. Much less consideration has been given to assessing structural resilience during sequential earthquake and tsunami multi-hazard events. Such events present a series of extreme loading scenarios, where damage sustained during the earthquake influences structural performance during the subsequent inundation. Similar difficulties exist with respect to damage sustained during tropical events, as wind and fluid loading may vary with structural response or accumulated damage. To help ensure critical structures meet a “life-safety” level of performance during such multi-hazard events, analysis software capable of simulating simultaneous structural and fluid dynamics must be developed. To address this gap in understanding of non-linear fluid-structure-interaction (FSI), an open-source tool (FOAMySees) was developed for simulation of tsunami and wave impact analysis of post-earthquake non-linear structural response of buildings. The tool is comprised of the Open-source Field Operation And Manipulation software package and OpenSeesPy, a Python 3 interpreter of OpenSees. The programs are coupledviapreCICE, a coupling library for partitioned multi-physics simulation. FOAMySees has been written to work in a Linux OS environment with HPC clusters in mind. The FOAMySees program offers a partitioned conventional-serial-staggered coupling scheme, with optional implicit iteration techniques to ensure a strongly-coupled two-way FSI solution. While FOAMySees was developed specifically for tsunami-resilience analysis, it may be utilized for other FSI applications with ease. With this coupled Computational Fluid Dynamics (CFD) and Finite Element Analysis (FEA) program, tsunami and earthquake simulations may be run sequentially or simultaneously, allowing for the evaluation of non-linear structural response to multi-hazard excitation.

     
    more » « less
  5. AWP-ODC is a 4th-order finite difference code used by the SCEC community for linear wave propagation, Iwan-type nonlinear dynamic rupture and wave propagation, and Strain Green Tensor simulation. We have ported and verified the CUDA-version of AWP-ODC-SGT, a reciprocal version used in the SCEC CyberShake project, to HIP so that it can also run on AMD GPUs. This code achieved sustained 32.6 Petaflop/s performance and 95.6% parallel efficiency at full scale on Frontier, a Leadership Computing Facility at Oak Ridge National Laboratory. The readiness of this community software on AMD Radeon Instinct GPUs and EPYC CPUs allows SCEC to take advantage of exascale systems to produce more realistic ground motions and accurate seismic hazard products. We have also deployed AWP-ODC to Azure to leverage the array of tools and services that Azure provides for tightly coupled HPC simulation on commercial cloud. We collaborated with Internet 2/Azure Accelerator supporting team, as part of Microsoft Internet2/Azure Accelerator for Research Fall 2022 Program, with Azure credits awarded through Cloudbank, an NSF-funded initiative. We demonstrate the AWP performance with a benchmark of ground motion simulation on various GPU based cloud instances, and a comparison of the cloud solution to on-premises bare-metal systems. AWP-ODC currently achieves excellent speedup and efficiency on CPU and GPU architectures. The Iwan-type dynamic rupture and wave propagation solver faces significant challenges, however, due to the increased computational workload with the number of yield surfaces chosen. Compared to linear solution, the Iwan model adds 10x-30x more computational time plus 5x-13x more memory consumption that require substantial code changes to obtain excellent performance. Supported by NSF’s Characteristic Science Applications (CSA) program for the Leadership-Class Computing Facility (LCCF) at Texas Advanced Computing Center (TACC), we are porting and improving the performance of this nonlinear AWP-ODC software, preparing for the next generation NSF LCCF system called Horizon, to be installed at TACC. During Texascale days on the current TACC’s Frontera, we carried out an Iwan-type nonlinear dynamic rupture and wave propagation simulation of a Mw7.8 scenario earthquake on the southern San Andreas fault. This simulation modeled 83 seconds of rupture with a grid spacing of 25 m to resolve frequencies up to 4 Hz with a minimum shear-wave velocity of 500 m/s. 
    more » « less