Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed for the detection and interpretation of gravitational waves. Future detectors will require even more accurate waveforms than those currently used. We present here the first large scale, adaptive mesh, multi-GPU numerical relativity (NR) code together with performance analysis and benchmarking. While comparisons are difficult to make, our GPU extension of the Dendro-GR NR code achieves a 6x speedup over existing state-of-the-art codes. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 8 A100s and weak scaling up to 229,376 ×86 cores on the Texas Advanced Computing Center's Frontera system.
more »
« less
A scalable framework for adaptive computational general relativity on heterogeneous clusters
We present a portable and highly-scalable framework that targets problems in the astrophysics and numerical relativity communities. This framework combines together the parallel Dendro octree with wavelet adaptive multiresolution and an automatic code-generation physics module to solve the Einstein equations of general relativity in the BSSNOK formulation. The goal of this work is to perform advanced, massively parallel numerical simulations of binary black hole and neutron star mergers, including Intermediate Mass Ratio Inspirals (IMRIs) of binary black holes with mass ratios on the order of 100:1. These studies will be used to study waveforms for use in LIGO data analysis and to calibrate approximate methods for generating gravitational waveforms. The key contribution of this work is the development of automatic code generators for computational relativity supporting SIMD vectorization, OpenMP, and CUDA combined with efficient distributed memory adaptive data-structures. These have enabled the development of efficient codes that demonstrate excellent weak scalability up to 131K cores on ORNL's Titan for binary mergers for mass ratios up to 100.
more »
« less
- NSF-PAR ID:
- 10106225
- Date Published:
- Journal Name:
- Proceedings of the ACM International Conference on Supercomputing ICS'19
- Page Range / eLocation ID:
- 1 to 12
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Study of the intermediate mass ratio black hole binary merger up to 1000:1 with numerical relativityAbstract We explicitly demonstrate that current numerical relativity techniques are able to accurately evolve black hole binaries with mass ratios of the order of 1000:1. This proof of principle is relevant for future third generation gravitational wave detectors and space mission LISA, as by purely numerical methods we would be able to accurately compute gravitational waves from the last stages of black hole mergers, as directly predicted by general relativity. We perform a sequence of simulations in the intermediate to small mass ratio regime, m 1 p / m 2 p = 1 / 7 , 1 / 16 , 1 / 32 , 1 / 64 , 1 / 128 , 1 / 256 , 1 / 512 , 1 / 1024 , with the small hole starting from rest at a proper distance D ≈ 13 M . We compare these headon full numerical evolutions with the corresponding semianalytic point particle perturbative results finding an impressive agreement for the total gravitational radiated energy and linear momentum as well as for the waveform spectra. We display numerical convergence of the results and identify the minimal numerical resolutions required to accurately solve for these very low amplitude gravitational waves. This work represents a first step towards the considerable challenge of applying numerical-relativity waveforms to interpreting gravitational-wave observations by LISA and next-generation ground-based gravitational-wave detectors.more » « less
-
more » « less
Abstract We evaluate several neural-network architectures, both convolutional and recurrent, for gravitational-wave time-series feature extraction by performing point parameter estimation on noisy waveforms from binary-black-hole mergers. We build datasets of 100 000 elements for each of four different waveform models (or approximants) in order to test how approximant choice affects feature extraction. Our choices include
SEOBNRv4P andIMRPhenomPv3 , which contain only the dominant quadrupole emission mode, alongsideIMRPhenomPv3HM andNRHybSur3dq8 , which also account for high-order modes. Each dataset element is injected into detector noise corresponding to the third observing run of the LIGO-Virgo-KAGRA (LVK) collaboration. We identify the temporal convolutional network architecture as the overall best performer in terms of training and validation losses and absence of overfitting to data. Comparison of results between datasets shows that the choice of waveform approximant for the creation of a dataset conditions the feature extraction ability of a trained network. Hence, care should be taken when building a dataset for the training of neural networks, as certain approximants may result in better network convergence of evaluation metrics. However, this performance does not necessarily translate to data which is more faithful to numerical relativity simulations. We also apply this network on actual signals from LVK runs, finding that its feature-extracting performance can be effective on real data. -
Intermediate-mass black holes (IMBHs) span the approximate mass range 100−10 5 M ⊙ , between black holes (BHs) that formed by stellar collapse and the supermassive BHs at the centers of galaxies. Mergers of IMBH binaries are the most energetic gravitational-wave sources accessible by the terrestrial detector network. Searches of the first two observing runs of Advanced LIGO and Advanced Virgo did not yield any significant IMBH binary signals. In the third observing run (O3), the increased network sensitivity enabled the detection of GW190521, a signal consistent with a binary merger of mass ∼150 M ⊙ providing direct evidence of IMBH formation. Here, we report on a dedicated search of O3 data for further IMBH binary mergers, combining both modeled (matched filter) and model-independent search methods. We find some marginal candidates, but none are sufficiently significant to indicate detection of further IMBH mergers. We quantify the sensitivity of the individual search methods and of the combined search using a suite of IMBH binary signals obtained via numerical relativity, including the effects of spins misaligned with the binary orbital axis, and present the resulting upper limits on astrophysical merger rates. Our most stringent limit is for equal mass and aligned spin BH binary of total mass 200 M ⊙ and effective aligned spin 0.8 at 0.056 Gpc −3 yr −1 (90% confidence), a factor of 3.5 more constraining than previous LIGO-Virgo limits. We also update the estimated rate of mergers similar to GW190521 to 0.08 Gpc −3 yr −1 .more » « less
-
Abstract We demonstrate the flexibility and utility of the Berger–Rigoutsos adaptive mesh refinement (AMR) algorithm used in the open-source numerical relativity (NR) code GRC hombo for generating gravitational waveforms from binary black-hole (BH) inspirals, and for studying other problems involving non-trivial matter configurations. We show that GRC hombo can produce high quality binary BH waveforms through a code comparison with the established NR code L ean . We also discuss some of the technical challenges involved in making use of full AMR (as opposed to, e.g. moving box mesh refinement), including the numerical effects caused by using various refinement criteria when regridding. We suggest several ‘rules of thumb’ for when to use different tagging criteria for simulating a variety of physical phenomena. We demonstrate the use of these different criteria through example evolutions of a scalar field theory. Finally, we also review the current status and general capabilities of GRC hombo .more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3330345.3330346
