Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed for the detection and interpretation of gravitational waves. Future detectors will require even more accurate waveforms than those currently used. We present here the first large scale, adaptive mesh, multi-GPU numerical relativity (NR) code together with performance analysis and benchmarking. While comparisons are difficult to make, our GPU extension of the Dendro-GR NR code achieves a 6x speedup over existing state-of-the-art codes. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 8 A100s and weak scaling up to 229,376 ×86 cores on the Texas Advanced Computing Center's Frontera system.
A scalable framework for adaptive computational general relativity on heterogeneous clusters
We present a portable and highly-scalable framework that targets problems in the astrophysics and numerical relativity communities. This framework combines together the parallel Dendro octree with wavelet adaptive multiresolution and an automatic code-generation physics module to solve the Einstein equations of general relativity in the BSSNOK formulation. The goal of this work is to perform advanced, massively parallel numerical simulations of binary black hole and neutron star mergers, including Intermediate Mass Ratio Inspirals (IMRIs) of binary black holes with mass ratios on the order of 100:1. These studies will be used to study waveforms for use in LIGO data analysis and to calibrate approximate methods for generating gravitational waveforms. The key contribution of this work is the development of automatic code generators for computational relativity supporting SIMD vectorization, OpenMP, and CUDA combined with efficient distributed memory adaptive data-structures. These have enabled the development of efficient codes that demonstrate excellent weak scalability up to 131K cores on ORNL's Titan for binary mergers for mass ratios up to 100.
- Publication Date:
- NSF-PAR ID:
- 10106225
- Journal Name:
- Proceedings of the ACM International Conference on Supercomputing ICS'19
- Page Range or eLocation-ID:
- 1 to 12
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Study of the intermediate mass ratio black hole binary merger up to 1000:1 with numerical relativityAbstract We explicitly demonstrate that current numerical relativity techniques are able to accurately evolve black hole binaries with mass ratios of the order of 1000:1. This proof of principle is relevant for future third generation gravitational wave detectors and space mission LISA, as by purely numerical methods we would be able to accurately compute gravitational waves from the last stages of black hole mergers, as directly predicted by general relativity. We perform a sequence of simulations in the intermediate to small mass ratio regime, m 1 p / m 2 p = 1 / 7 , 1 / 16 , 1 / 32 , 1 / 64 , 1 / 128 , 1 / 256 , 1 / 512 , 1 / 1024 , with the small hole starting from rest at a proper distance D ≈ 13 M . We compare these headon full numerical evolutions with the corresponding semianalytic point particle perturbative results finding an impressive agreement for the total gravitational radiated energy and linear momentum as well as for the waveform spectra. We display numerical convergence of the results and identify the minimal numerical resolutions required to accurately solve for these very low amplitude gravitational waves. Thismore »
-
Intermediate-mass black holes (IMBHs) span the approximate mass range 100−10 5 M ⊙ , between black holes (BHs) that formed by stellar collapse and the supermassive BHs at the centers of galaxies. Mergers of IMBH binaries are the most energetic gravitational-wave sources accessible by the terrestrial detector network. Searches of the first two observing runs of Advanced LIGO and Advanced Virgo did not yield any significant IMBH binary signals. In the third observing run (O3), the increased network sensitivity enabled the detection of GW190521, a signal consistent with a binary merger of mass ∼150 M ⊙ providing direct evidence of IMBH formation. Here, we report on a dedicated search of O3 data for further IMBH binary mergers, combining both modeled (matched filter) and model-independent search methods. We find some marginal candidates, but none are sufficiently significant to indicate detection of further IMBH mergers. We quantify the sensitivity of the individual search methods and of the combined search using a suite of IMBH binary signals obtained via numerical relativity, including the effects of spins misaligned with the binary orbital axis, and present the resulting upper limits on astrophysical merger rates. Our most stringent limit is for equal mass and alignedmore »
-
Abstract We demonstrate the flexibility and utility of the Berger–Rigoutsos adaptive mesh refinement (AMR) algorithm used in the open-source numerical relativity (NR) code GRC hombo for generating gravitational waveforms from binary black-hole (BH) inspirals, and for studying other problems involving non-trivial matter configurations. We show that GRC hombo can produce high quality binary BH waveforms through a code comparison with the established NR code L ean . We also discuss some of the technical challenges involved in making use of full AMR (as opposed to, e.g. moving box mesh refinement), including the numerical effects caused by using various refinement criteria when regridding. We suggest several ‘rules of thumb’ for when to use different tagging criteria for simulating a variety of physical phenomena. We demonstrate the use of these different criteria through example evolutions of a scalar field theory. Finally, we also review the current status and general capabilities of GRC hombo .
-
Abstract Numerical relativity (NR) simulations of binary black hole (BBH) systems provide the most accurate gravitational wave predictions, but at a high computational cost—especially when the black holes have nearly extremal spins (i.e. spins near the theoretical upper limit) or very unequal masses. Recently, the technique of reduced order modeling has enabled the construction of ‘surrogate models’ trained on an existing set of NR waveforms. Surrogate models enable the rapid computation of the gravitational waves emitted by BBHs. Typically these models are used for interpolation to compute gravitational waveforms for BBHs with mass ratios and spins within the bounds of the training set. Because simulations with nearly extremal spins are so technically challenging, surrogate models almost always rely on training sets with only moderate spins. In this paper, we explore how well surrogate models can extrapolate to nearly extremal spins when the training set only includes moderate spins. For simplicity, we focus on one-dimensional surrogate models trained on NR simulations of BBHs with equal masses and equal, aligned spins. We assess the performance of the surrogate models at higher spin magnitudes by calculating the mismatches between extrapolated surrogate model waveforms and NR waveforms, by calculating the differences between extrapolated andmore »
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Publisher's Version of Record
- https://doi.org/10.1145/3330345.3330346
