skip to main content

Title: GRaM-X: a new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit

We presentGRaM-X(GeneralRelativisticacceleratedMagnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems.GRaM-Xsupports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit calledCarpetXwhich in turn leveragesAMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD.GRaM-Xsupports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find thatGRaM-Xshows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit.

more » « less
Author(s) / Creator(s):
; ; ; ; ;
Publisher / Repository:
IOP Publishing
Date Published:
Journal Name:
Classical and Quantum Gravity
Page Range / eLocation ID:
Article No. 205009
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Chi-Wang Shu (Ed.)
    GPU computing is expected to play an integral part in all modern Exascale supercomputers. It is also expected that higher order Godunov schemes will make up about a significant fraction of the application mix on such supercomputers. It is, therefore, very important to prepare the community of users of higher order schemes for hyperbolic PDEs for this emerging opportunity. Not every algorithm that is used in the space-time update of the solution of hyperbolic PDEs will take well to GPUs. However, we identify a small core of algorithms that take exceptionally well to GPU computing. Based on an analysis of available options, we have been able to identify weighted essentially non-oscillatory (WENO) algorithms for spatial reconstruction along with arbitrary derivative (ADER) algorithms for time extension followed by a corrector step as the winning three-part algorithmic combination. Even when a winning subset of algorithms has been identified, it is not clear that they will port seamlessly to GPUs. The low data throughput between CPU and GPU, as well as the very small cache sizes on modern GPUs, implies that we have to think through all aspects of the task of porting an application to GPUs. For that reason, this paper identifies the techniques and tricks needed for making a successful port of this very useful class of higher order algorithms to GPUs. Application codes face a further challenge—the GPU results need to be practically indistinguishable from the CPU results—in order for the legacy knowledge bases embedded in these applications codes to be preserved during the port of GPUs. This requirement often makes a complete code rewrite impossible. For that reason, it is safest to use an approach based on OpenACC directives, so that most of the code remains intact (as long as it was originally well-written). This paper is intended to be a one-stop shop for anyone seeking to make an OpenACC-based port of a higher order Godunov scheme to GPUs. We focus on three broad and high-impact areas where higher order Godunov schemes are used. The first area is computational fluid dynamics (CFD). The second is computational magnetohydrodynamics (MHD) which has an involution constraint that has to be mimetically preserved. The third is computational electrodynamics (CED) which has involution constraints and also extremely stiff source terms. Together, these three diverse uses of higher order Godunov methodology, cover many of the most important applications areas. In all three cases, we show that the optimal use of algorithms, techniques, and tricks, along with the use of OpenACC, yields superlative speedups on GPUs. As a bonus, we find a most remarkable and desirable result: some higher order schemes, with their larger operations count per zone, show better speedup than lower order schemes on GPUs. In other words, the GPU is an optimal stratagem for overcoming the higher computational complexities of higher order schemes. Several avenues for future improvement have also been identified. A scalability study is presented for a real-world application using GPUs and comparable numbers of high-end multicore CPUs. It is found that GPUs offer a substantial performance benefit over comparable number of CPUs, especially when all the methods designed in this paper are used. 
    more » « less
  2. Abstract

    We present the implementation of a two-moment-based general-relativistic multigroup radiation transport module in theGeneral-relativisticmultigridnumerical (Gmunu) code. On top of solving the general-relativistic magnetohydrodynamics and the Einstein equations with conformally flat approximations, the code solves the evolution equations of the zeroth- and first-order moments of the radiations in the Eulerian-frame. An analytic closure relation is used to obtain the higher order moments and close the system. The finite-volume discretization has been adopted for the radiation moments. The advection in spatial space and frequency-space are handled explicitly. In addition, the radiation–matter interaction terms, which are very stiff in the optically thick region, are solved implicitly. The implicit–explicit Runge–Kutta schemes are adopted for time integration. We test the implementation with a number of numerical benchmarks from frequency-integrated to frequency-dependent cases. Furthermore, we also illustrate the astrophysical applications in hot neutron star and core-collapse supernovae modelings, and compare with other neutrino transport codes.

    more » « less
  3. Abstract We use the public code ebhlight to carry out 3D radiative general relativistic magnetohydrodynamics (GRMHD) simulations of accretion on to the supermassive black hole in M87. The simulations self-consistently evolve a frequency-dependent Monte Carlo description of the radiation field produced by the accretion flow. We explore two limits of accumulated magnetic flux at the black hole (SANE and MAD), each coupled to several subgrid prescriptions for electron heating that are motivated by models of turbulence and magnetic reconnection. We present convergence studies for the radiation field and study its properties. We find that the near-horizon photon energy density is an order of magnitude higher than is predicted by simple isotropic estimates from the observed luminosity. The radially dependent photon momentum distribution is anisotropic and can be modeled by a set of point-sources near the equatorial plane. We draw properties of the radiation and magnetic field from the simulation and feed them into an analytic model of gap acceleration to estimate the very high energy (VHE) γ-ray luminosity from the magnetized jet funnel, assuming that a gap is able to form. We find luminosities of $\rm \sim 10^{41} \, erg \, s^{-1}$ for MAD models and $\rm \sim 2\times 10^{40} \, erg \, s^{-1}$ for SANE models, which are comparable to measurements of M87’s VHE flares. The time-dependence seen in our calculations is insufficient to explain the flaring behaviour. Our results provide a step towards bridging theoretical models of near-horizon properties seen in black hole images with the VHE activity of M87. 
    more » « less
  4. ABSTRACT Sgr A* exhibits regular variability in its multiwavelength emission, including daily X-ray flares and roughly continuous near-infrared (NIR) flickering. The origin of this variability is still ambiguous since both inverse Compton and synchrotron emission are possible radiative mechanisms. The underlying particle distributions are also not well constrained, particularly the non-thermal contribution. In this work, we employ the GPU-accelerated general relativistic magnetohydrodynamics code H-AMR to perform a study of flare flux distributions, including the effect of particle acceleration for the first time in high-resolution 3D simulations of Sgr A*. For the particle acceleration, we use the general relativistic ray-tracing code bhoss to perform the radiative transfer, assuming a hybrid thermal+non-thermal electron energy distribution. We extract ∼60 h light curves in the sub-millimetre, NIR and X-ray wavebands, and compare the power spectra and the cumulative flux distributions of the light curves to statistical descriptions for Sgr A* flares. Our results indicate that non-thermal populations of electrons arising from turbulence-driven reconnection in weakly magnetized accretion flows lead to moderate NIR and X-ray flares and reasonably describe the X-ray flux distribution while fulfilling multiwavelength flux constraints. These models exhibit high rms per cent amplitudes, $\gtrsim 150{{\ \rm per\ cent}}$ both in the NIR and the X-rays, with changes in the accretion rate driving the 230 GHz flux variability, in agreement with Sgr A* observations. 
    more » « less
  5. Abstract

    We perform a full 3D general relativistic magnetohydrodynamical (GRMHD) simulation of an equal-mass, spinning, binary black hole approaching merger, surrounded by a circumbinary disk and with a minidisk around each black hole. For this purpose, we evolve the ideal GRMHD equations on top of an approximated spacetime for the binary that is valid in every position of space, including the black hole horizons, during the inspiral regime. We use relaxed initial data for the circumbinary disk from a previous long-term simulation, where the accretion is dominated by am= 1 overdensity called the lump. We compare our new spinning simulation with a previous non-spinning run, studying how spin influences the minidisk properties. We analyze the accretion from the inner edge of the lump to the black hole, focusing on the angular momentum budget of the fluid around the minidisks. We find that minidisks in the spinning case have more mass over a cycle than the non-spinning case. However, in both cases we find that most of the mass received by the black holes is delivered by the direct plunging of material from the lump. We also analyze the morphology and variability of the electromagnetic fluxes, and we find they share the same periodicities of the accretion rate. In the spinning case, we find that the outflows are stronger than the non-spinning case. Our results will be useful to understand and produce realistic synthetic light curves and spectra, which can be used in future observations.

    more » « less