Search for: All records

Modern gravitational-wave science demands increasingly accurate and computationally intensive numerical relativity (NR) simulations. The Python-based, open-sourceNRPyframework generates optimized C/C++ code for NR, including the complete NR codeBlackHoles@Home(BH@H), which leverages curvilinear coordinates well-suited to many astrophysical scenarios. Historically,BH@Hwas limited to single-nodeOpenMPCPU parallelism. To address this, we introducesuperB, an open-source extension toNRPythat enables automatic generation of scalable, task-based, distributed-memoryCharm++code from existingBH@Hmodules. The generated code partitions the structured grids used byNRPy/BH@H, managing communication between them. Its correctness is validated through bit-identical results with the standardOpenMPversion on a single node and via a head-on binary black hole simulation in cylindrical-like coordinates, accurately reproducing quasi-normal modes (up to $\ell =8$). ThesuperB/NRPy-generated code demonstrates excellent strong scaling, achieving an ≈45× speedup on 64 nodes (7168 cores) compared to the original single-nodeOpenMPcode for a large 3D vacuum test. This scalable infrastructure benefits demanding simulations and lays the groundwork for future multi-patch grid support, targeting long inspirals, extreme parameter studies, and rapid follow-ups. This infrastructure readily integrates with otherNRPy/BH@H-based projects, enabling performant scaling for the general relativistic hydrodynamics codeGRoovy, and facilitating future coupling with GPU acceleration via theNRPy-CUDAproject. 

Next-generation gravitational wave (GW) detectors such as Cosmic Explorer, the Einstein Telescope, and LISA, demand highly accurate and extensive GW catalogs to faithfully extract physical parameters from observed signals. However, numerical relativity (NR) faces significant challenges in generating these catalogs at the required scale and accuracy on modern computers, as NR codes do not fully exploit modern GPU capabilities. In response, we extend NRPy, a Python-based NR code-generation framework, to develop NRPyEllipticGPU—a CUDA-optimized elliptic solver tailored for the binary black hole initial data problem. NRPyEllipticGPU is the first GPU-enabled elliptic solver in the NR community, supporting a variety of coordinate systems and demonstrating substantial performance improvements on both consumer-grade and HPC-grade GPUs. We show that, when compared to a high-end CPU, NRPyEllipticGPU achieves on a high-end GPU up to a sixteenfold speedup in single precision while increasing double-precision performance by a factor of 2–4. This performance boost leverages the GPU’s superior parallelism and memory bandwidth to achieve a compute-bound application and enhancing the overall simulation efficiency. As NRPyEllipticGPU shares the core infrastructure common to NR codes, this work serves as a practical guide for developing full, CUDA-optimized NR codes. 

Two-dimensional models assuming axisymmetry are an economical way to explore the long-term evolution of black hole accretion disks, but they are only realistic if the feedback of the nonaxisymmetric turbulence on the mean momentum and magnetic fields is incorporated. Dynamo terms added to the 2D induction equation should be calibrated to 3D magnetohydrodynamics simulations. For generality, the dynamo tensors should be calibrated as functions of local variables rather than explicit functions of spatial coordinates in a particular basis. In this paper, we study the feedback of nonaxisymmetric features on the 2D mean fields using a global 3D, relativistic, Cartesian simulation from the illinoisgrmhd code. We introduce new methods for estimating overall dynamo alpha and turbulent diffusivity effects, as well as measures of the dominance of nonaxisymmetric components of energies and fluxes within the disk interior. We attempt closure models of the dynamo electromotive force using least-squares fitting, considering both models where coefficient tensors are functions of space and more global, covariant models. None of these models are judged satisfactory, but we are able to draw conclusions on what sorts of generalizations are and are not promising.