Search for: All records

Abstract We presentAsterX, a novel open-source, modular, GPU-accelerated, fully general relativistic magnetohydrodynamic (GRMHD) code designed for dynamic spacetimes in 3D Cartesian coordinates, and tailored for exascale computing. We utilize block-structured adaptive mesh refinement (AMR) throughCarpetX, the new driver for theEinstein Toolkit, which is built onAMReX, a software framework for massively parallel applications.AsterXemploys the Valencia formulation for GRMHD, coupled with the ‘Z4c’ formalism for spacetime evolution, while incorporating high resolution shock capturing schemes to accurately handle the hydrodynamics.AsterXhas undergone rigorous testing in both static and dynamic spacetime, demonstrating remarkable accuracy and agreement with other codes in literature. Using subcycling in time, we find an overall performance gain of factor 2.5–4.5. Benchmarking the code through scaling tests on OLCF’s Frontier supercomputer, we demonstrate a weak scaling efficiency of about 67%–77% on 4096 nodes compared to an 8-node performance. 

Triaxial neutron stars can be sources of continuous gravitational radiation detectable by ground-based interferometers. The amplitude of the emitted gravitational wave can be greatly affected by the state of the hydrodynamical fluid flow inside the neutron star. In this work, we examine the most triaxial models along two sequences of constant rest mass, confirming their dynamical stability. We also study the response of a triaxial figure of quasiequilibrium under a variety of perturbations that lead to different fluid flows. Starting from the general relativistic compressible analog of the Newtonian Jacobi ellipsoid, we perform simulations of Dedekind-type flows. We find that in some cases the triaxial neutron star resembles a Riemann-S-type ellipsoid with minor rotation and gravitational wave emission as it evolves towards axisymmetry. The present results highlight the importance of understanding the fluid flow in the interior of a neutron star in terms of its gravitational wave content. 

The Einstein Toolkit is a complex software system for numerical general relativity, a science domain that includes colliding black holes, neutron stars, supernovae, etc. As might be expected for a framework of this size and age (parts of it are over 20 years old), there is a significant learning curve to building it, running it, writing new modules for it, etc. Over the years, the Einstein Toolkit maintainers have given a number of tutorials for new users. In recent years, we have created a tutorial server which allows us to streamline the teaching/learning process through the use of Jupyter notebooks and docker images. In this paper we describe the special considerations and adaptations required by the image and the notebook server that enable us to (1) easily make logins and manage accounts which streamlines both the classroom and the independent study experiences, (2) create a simplified but natural user experience for compiling and developing a complex C++ application, (3) scale to increasing class sizes. 

Reproducibility of results is a cornerstone of the scientific method. Scientific computing encounters two challenges when aiming for this goal. Firstly, reproducibility should not depend on details of the runtime environment, such as the compiler version or computing environment, so results are verifiable by third-parties. Secondly, different versions of software code executed in the same runtime environment should produce consistent numerical results for physical quantities. In this manuscript, we test the feasibility of reproducing scientific results obtained using the IllinoisGRMHD code that is part of an open-source community software for simulation in relativistic astrophysics, the Einstein Toolkit. We verify that numerical results of simulating a single isolated neutron star with IllinoisGRMHD can be reproduced, and compare them to results reported by the code authors in 2015. We use two different supercomputers: Expanse at SDSC, and Stampede2 at TACC. By compiling the source code archived along with the paper on both Expanse and Stampede2, we find that IllinoisGRMHD reproduces results published in its announcement paper up to errors comparable to round-off level changes in initial data parameters. We also verify that a current version of IllinoisGRMHD reproduces these results once we account for bug fixes which have occurred since the original publication. 

We present GRaM-X (General Relativistic accelerated Magnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems. GRaM-X supports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit called CarpetX which in turn leverages AMReX, an AMR library developed for use by the United States DOE's Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD. GRaM-X supports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find that GRaM-X shows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF's supercomputer Summit. 

Abstract We present a 3D general-relativistic magnetohydrodynamic simulation of a short-lived neutron star remnant formed in the aftermath of a binary neutron star merger. The simulation uses an M1 neutrino transport scheme to track neutrino–matter interactions and is well suited to studying the resulting nucleosynthesis and kilonova emission. A magnetized wind is driven from the remnant and ejects neutron-rich material at a quasi-steady-state rate of 0.8 × 10⁻¹M_⊙s⁻¹. We find that the ejecta in our simulations underproducer-process abundances beyond the secondr-process peak. For sufficiently long-lived remnants, these outflowsalonecan produce blue kilonovae, including the blue kilonova component observed for AT2017gfo. 

Abstract We presentGRaM-X(GeneralRelativisticacceleratedMagnetohydrodynamics on AMReX), a new GPU-accelerated dynamical-spacetime general relativistic magnetohydrodynamics (GRMHD) code which extends the GRMHD capability of Einstein Toolkit to GPU-based exascale systems.GRaM-Xsupports 3D adaptive mesh refinement (AMR) on GPUs via a new AMR driver for the Einstein Toolkit calledCarpetXwhich in turn leveragesAMReX, an AMR library developed for use by the United States DOE’s Exascale Computing Project. We use the Z4c formalism to evolve the Einstein equations and the Valencia formulation to evolve the equations of GRMHD.GRaM-Xsupports both analytic as well as tabulated equations of state. We implement TVD and WENO reconstruction methods as well as the HLLE Riemann solver. We test the accuracy of the code using a range of tests on static spacetime, e.g. 1D magnetohydrodynamics shocktubes, the 2D magnetic rotor and a cylindrical explosion, as well as on dynamical spacetimes, i.e. the oscillations of a 3D Tolman-Oppenheimer-Volkhof star. We find excellent agreement with analytic results and results of other codes reported in literature. We also perform scaling tests and find thatGRaM-Xshows a weak scaling efficiency of ∼40%–50% on 2304 nodes (13824 NVIDIA V100 GPUs) with respect to single-node performance on OLCF’s supercomputer Summit. 

ABSTRACT We present a new moment-based energy-integrated neutrino transport code for neutron star merger simulations in general relativity. In the merger context, ours is the first code to include Doppler effects at all orders in υ/c, retaining all non-linear neutrino–matter coupling terms. The code is validated with a stringent series of tests. We show that the inclusion of full neutrino–matter coupling terms is necessary to correctly capture the trapping of neutrinos in relativistically moving media, such as in differentially rotating merger remnants. We perform preliminary simulations proving the robustness of the scheme in simulating ab-initio mergers to black hole collapse and long-term neutron star remnants up to $${\sim }70\,$$ ms. The latter is the longest dynamical space-time, 3D, general relativistic simulations with full neutrino transport to date. We compare results obtained at different resolutions and using two different closures for the moment scheme. We do not find evidences of significant out-of-thermodynamic equilibrium effects, such as bulk viscosity, on the post-merger dynamics or gravitational wave emission. Neutrino luminosities and average energies are in good agreement with theory expectations and previous simulations by other groups using similar schemes. We compare dynamical and early wind ejecta properties obtained with M1 and with our older neutrino treatment. We find that the M1 results have systematically larger proton fractions. However, the differences in the nucleosynthesis yields are modest. This work sets the basis for future detailed studies spanning a wider set of neutrino reactions, binaries, and equations of state.