skip to main content

Title: Lessons for adaptive mesh refinement in numerical relativity
Abstract We demonstrate the flexibility and utility of the Berger–Rigoutsos adaptive mesh refinement (AMR) algorithm used in the open-source numerical relativity (NR) code GRC hombo for generating gravitational waveforms from binary black-hole (BH) inspirals, and for studying other problems involving non-trivial matter configurations. We show that GRC hombo can produce high quality binary BH waveforms through a code comparison with the established NR code L ean . We also discuss some of the technical challenges involved in making use of full AMR (as opposed to, e.g. moving box mesh refinement), including the numerical effects caused by using various refinement criteria when regridding. We suggest several ‘rules of thumb’ for when to use different tagging criteria for simulating a variety of physical phenomena. We demonstrate the use of these different criteria through example evolutions of a scalar field theory. Finally, we also review the current status and general capabilities of GRC hombo .  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ; ; ; ; ; ; ;
Date Published:
Journal Name:
Classical and Quantum Gravity
Page Range / eLocation ID:
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    ABSTRACT The morphology of bipolar planetary nebulae (PNe) can be attributed to interactions between a fast wind from the central engine and the dense toroidal-shaped ejecta left over from common envelope (CE) evolution. Here we use the 3D hydrodynamic adaptive mesh refinement (AMR) code AstroBEAR to study the possibility that bipolar PN outflows can emerge collimated even from an uncollimated spherical wind in the aftermath of a CE event. The output of a single CE simulation via the smoothed particle hydrodynamics (SPH) code phantom serves as the initial conditions. Four cases of winds, all with high enough momenta to account for observed high momenta pre-PN outflows, are injected spherically from the region of the CE binary remnant into the ejecta. We compare cases with two different momenta and cases with no radiative cooling versus application of optically thin emission via a cooling curve to the outflow. Our simulations show that in all cases highly collimated bipolar outflows result from deflection of the spherical wind via the interaction with the CE ejecta. Significant asymmetries between the top and bottom lobes are seen in all cases. The asymmetry is strongest for the lower momentum case with radiative cooling. While real post-CE winds may be aspherical, our models show that collimation via ‘inertial confinement’ will be strong enough to create jet-like outflows even beginning with maximally uncollimated drivers. Our simulations reveal detailed shock structures in the shock-focused inertial confinement (SFIC) model and develop a lens-shaped inner shock that is a new feature of SFIC-driven bipolar lobes. 
    more » « less
  2. Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed for the detection and interpretation of gravitational waves. Future detectors will require even more accurate waveforms than those currently used. We present here the first large scale, adaptive mesh, multi-GPU numerical relativity (NR) code together with performance analysis and benchmarking. While comparisons are difficult to make, our GPU extension of the Dendro-GR NR code achieves a 6x speedup over existing state-of-the-art codes. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 8 A100s and weak scaling up to 229,376 ×86 cores on the Texas Advanced Computing Center's Frontera system. 
    more » « less
  3. Abstract Numerical relativity (NR) simulations of binary black hole (BBH) systems provide the most accurate gravitational wave predictions, but at a high computational cost—especially when the black holes have nearly extremal spins (i.e. spins near the theoretical upper limit) or very unequal masses. Recently, the technique of reduced order modeling has enabled the construction of ‘surrogate models’ trained on an existing set of NR waveforms. Surrogate models enable the rapid computation of the gravitational waves emitted by BBHs. Typically these models are used for interpolation to compute gravitational waveforms for BBHs with mass ratios and spins within the bounds of the training set. Because simulations with nearly extremal spins are so technically challenging, surrogate models almost always rely on training sets with only moderate spins. In this paper, we explore how well surrogate models can extrapolate to nearly extremal spins when the training set only includes moderate spins. For simplicity, we focus on one-dimensional surrogate models trained on NR simulations of BBHs with equal masses and equal, aligned spins. We assess the performance of the surrogate models at higher spin magnitudes by calculating the mismatches between extrapolated surrogate model waveforms and NR waveforms, by calculating the differences between extrapolated and NR measurements of the remnant black-hole mass, and by testing how the surrogate model improves as the training set extends to higher spins. We find that while extrapolation in this one-dimensional case is viable for current detector sensitivities, surrogate models for next-generation detectors should use training sets that extend to nearly extremal spins. 
    more » « less
  4. Abstract

    We presentGRaM-X(GeneralRelativisticacceleratedMagnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems.GRaM-Xsupports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit calledCarpetXwhich in turn leveragesAMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD.GRaM-Xsupports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find thatGRaM-Xshows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit.

    more » « less

    Strong dynamical interactions among stars and compact objects are expected in a variety of astrophysical settings, such as star clusters and the disks of active galactic nuclei. Via a suite of three-dimensional hydrodynamics simulations using the moving-mesh code arepo, we investigate the formation of transient phenomena and their properties in close encounters between an $2\, {\rm M}_{\odot }$ or $20\, {\rm M}_{\odot }$ equal-mass circular binary star and single $20\, {\rm M}_{\odot }$ black hole (BH). Stars can be disrupted by the BH during dynamical interactions, naturally producing electromagnetic transient phenomena. Encounters with impact parameters smaller than the semimajor axis of the initial binary frequently lead to a variety of transients whose electromagnetic signatures are qualitatively different from those of ordinary disruption events involving just two bodies. These include the simultaneous or successive disruptions of both stars and one full disruption of one star accompanied by successive partial disruptions of the other star. On the contrary, when the impact parameter is larger than the semimajor axis of the initial binary, the binary is either simply tidally perturbed or dissociated into bound and unbound single stars (‘micro-Hills’ mechanism). The dissociation of $20\, {\rm M}_{\odot }$ binaries can produce a runaway star and an active BH moving away from one another. Also, the binary dissociation can either produce an interacting binary with the BH, or a non-interacting, hard binary; both could be candidates of BH high- and low-mass X-ray binaries. Hence, our simulations especially confirm that strong encounters can lead to the formation of the (generally difficult to form) BH low-mass X-ray binaries.

    more » « less