More Like this

1. GRaM-X: a new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

   https://doi.org/10.1088/1361-6382/acf2d9  

   Shankar, Swapnil; Mösta, Philipp; Brandt, Steven R.; Haas, Roland; Schnetter, Erik; de Graaf, Yannick (September 2023, Classical and Quantum Gravity)

   Abstract We presentGRaM-X(GeneralRelativisticacceleratedMagnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems.GRaM-Xsupports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit calledCarpetXwhich in turn leveragesAMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD.GRaM-Xsupports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find thatGRaM-Xshows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit. 

   more » « less
2. AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes

   https://doi.org/10.1088/1361-6382/ad9c11  

   Kalinani, Jay V; Ji, Liwei; Ennoggi, Lorenzo; Lopez_Armengol, Federico G; Timotheo_Sanches, Lucas; Tsao, Bing-Jyun; Brandt, Steven R; Campanelli, Manuela; Ciolfi, Riccardo; Giacomazzo, Bruno; et al (December 2024, Classical and Quantum Gravity)

   Abstract We presentAsterX, a novel open-source, modular, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) throughCarpetX, the new driver for theEinstein Toolkit, which is built onAMReX, a software framework for massively parallel applications.AsterXemploys the Valencia formulation for GRMHD, coupled with the ‘Z4c’ formalism for spacetime evolution, while incorporating high resolution shock capturing schemes to accurately handle the hydrodynamics.AsterXhas undergone rigorous testing in both static and dynamic spacetime, demonstrating remarkable accuracy and agreement with other codes in literature. Using subcycling in time, we find an overall performance gain of factor 2.5–4.5. Benchmarking the code through scaling tests on OLCF’s Frontier supercomputer, we demonstrate a weak scaling efficiency of about 67%–77% on 4096 nodes compared to an 8-node performance. 

   more » « less
3. Large-scale poloidal magnetic field dynamo leads to powerful jets in GRMHD simulations of black hole accretion with toroidal field

   https://doi.org/10.1093/mnras/staa955  

   Liska, M; Tchekhovskoy, A; Quataert, E (May 2020, Monthly Notices of the Royal Astronomical Society)

   ABSTRACT Accreting black holes (BHs) launch relativistic collimated jets, across many decades in luminosity and mass, suggesting the jet launching mechanism is universal, robust, and scale-free. Theoretical models and general relativistic magnetohydrodynamic (GRMHD) simulations indicate that the key jet-making ingredient is large-scale poloidal magnetic flux. However, its origin is uncertain, and it is unknown if it can be generated in situ or dragged inward from the ambient medium. Here, we use the GPU-accelerated GRMHD code h-amr to study global 3D BH accretion at unusually high resolutions more typical of local shearing box simulations. We demonstrate that turbulence in a radially extended accretion disc can generate large-scale poloidal magnetic flux in situ, even when starting from a purely toroidal magnetic field. The flux accumulates around the BH till it becomes dynamically important, leads to a magnetically arrested disc (MAD), and launches relativistic jets that are more powerful than the accretion flow. The jet power exceeds that of previous GRMHD toroidal field simulations by a factor of 10 000. The jets do not show significant kink or pinch instabilities, accelerate to γ ∼ 10 over three decades in distance, and follow a collimation profile similar to the observed M87 jet. 

   more » « less
4. H-AMR: A New GPU-accelerated GRMHD Code for Exascale Computing with 3D Adaptive Mesh Refinement and Local Adaptive Time Stepping

   https://doi.org/10.3847/1538-4365/ac9966  

   Liska, M_T P; Chatterjee, K; Issa, D; Yoon, D; Kaaz, N; Tchekhovskoy, A; van_Eijnatten, D; Musoke, G; Hesp, C; Rohoza, V; et al (November 2022, The Astrophysical Journal Supplement Series)

   Abstract General relativistic magnetohydrodynamic (GRMHD) simulations have revolutionized our understanding of black hole accretion. Here, we present a GPU-accelerated GRMHD code H-AMR with multifaceted optimizations that, collectively, accelerate computation by 2–5 orders of magnitude for a wide range of applications. First, it introduces a spherical grid with 3D adaptive mesh refinement that operates in each of the three dimensions independently. This allows us to circumvent the Courant condition near the polar singularity, which otherwise cripples high-resolution computational performance. Second, we demonstrate that local adaptive time stepping on a logarithmic spherical-polar grid accelerates computation by a factor of ≲10 compared to traditional hierarchical time-stepping approaches. Jointly, these unique features lead to an effective speed of ∼10⁹zone cycles per second per node on 5400 NVIDIA V100 GPUs (i.e., 900 nodes of the OLCF Summit supercomputer). We illustrate H-AMR's computational performance by presenting the first GRMHD simulation of a tilted thin accretion disk threaded by a toroidal magnetic field around a rapidly spinning black hole. With an effective resolution of 13,440 × 4608 × 8092 cells and a total of ≲22 billion cells and ∼0.65 × 10⁸time steps, it is among the largest astrophysical simulations ever performed. We find that frame dragging by the black hole tears up the disk into two independently precessing subdisks. The innermost subdisk rotation axis intermittently aligns with the black hole spin, demonstrating for the first time that such long-sought alignment is possible in the absence of large-scale poloidal magnetic fields. 

   more » « less
5. General relativistic MHD simulations of non-thermal flaring in Sagittarius A*

   https://doi.org/10.1093/mnras/stab2466  

   Chatterjee, K; Markoff, S; Neilsen, J; Younsi, Z; Witzel, G; Tchekhovskoy, A; Yoon, D; Ingram, A; van der Klis, M; Boyce, H; et al (September 2021, Monthly Notices of the Royal Astronomical Society)

   ABSTRACT Sgr A* exhibits regular variability in its multiwavelength emission, including daily X-ray flares and roughly continuous near-infrared (NIR) flickering. The origin of this variability is still ambiguous since both inverse Compton and synchrotron emission are possible radiative mechanisms. The underlying particle distributions are also not well constrained, particularly the non-thermal contribution. In this work, we employ the GPU-accelerated general relativistic magnetohydrodynamics code H-AMR to perform a study of flare flux distributions, including the effect of particle acceleration for the first time in high-resolution 3D simulations of Sgr A*. For the particle acceleration, we use the general relativistic ray-tracing code bhoss to perform the radiative transfer, assuming a hybrid thermal+non-thermal electron energy distribution. We extract ∼60 h light curves in the sub-millimetre, NIR and X-ray wavebands, and compare the power spectra and the cumulative flux distributions of the light curves to statistical descriptions for Sgr A* flares. Our results indicate that non-thermal populations of electrons arising from turbulence-driven reconnection in weakly magnetized accretion flows lead to moderate NIR and X-ray flares and reasonably describe the X-ray flux distribution while fulfilling multiwavelength flux constraints. These models exhibit high rms per cent amplitudes, $$\gtrsim 150{{\ \rm per\ cent}}$$ both in the NIR and the X-rays, with changes in the accretion rate driving the 230 GHz flux variability, in agreement with Sgr A* observations. 

   more » « less