More Like this

1. Simulating binary black hole mergers using discontinuous Galerkin methods

   https://doi.org/10.1088/1361-6382/ad9f19  

   Lovelace, Geoffrey; Nelli, Kyle_C; Deppe, Nils; Vu, Nils_L; Throwe, William; Bonilla, Marceline_S; Carpenter, Alexander; Kidder, Lawrence_E; Macedo, Alexandra; Scheel, Mark_A; et al (January 2025, Classical and Quantum Gravity)

   Abstract Binary black holes are the most abundant source of gravitational-wave observations. Gravitational-wave observatories in the next decade will require tremendous increases in the accuracy of numerical waveforms modeling binary black holes, compared to today’s state of the art. One approach to achieving the required accuracy is using spectral-type methods that scale to many processors. Using theSpECTREnumerical-relativity (NR) code, we present the first simulations of a binary black hole inspiral, merger, and ringdown using discontinuous Galerkin (DG) methods. The efficiency of DG methods allows us to evolve the binary through ∼ 18 orbits at reasonable computational cost. We then useSpECTRE’s Cauchy Characteristic Evolution (CCE) code to extract the gravitational waves at future null infinity. The open-source nature ofSpECTREmeans this is the first time a spectral-type method for simulating binary black hole evolutions is available to the entire NR community. 

   more » « less
2. Comparison of adaptive mesh refinement techniques for numerical weather prediction

   Abdi, Daniel S; Almgren, Ann; Giraldo, Francis X; Jankov, Isidora (April 2024, arXiv but under review in Monthly Weather Review)

   This paper examines the application of adaptive mesh refinement (AMR) in the field of numerical weather prediction (NWP). We implement and assess two distinct AMR approaches and evaluate their performance through standard NWP benchmarks. In both cases, we solve the fully compressible Euler equations, fundamental to many non-hydrostatic weather models. The first approach utilizes oct-tree cell-based mesh refinement coupled with a high-order discontinuous Galerkin method for spatial discretization. In the second approach, we employ level-based AMR with the finite difference method. Our study provides insights into the accuracy and benefits of employing these AMR methodologies for the multi-scale problem of NWP. Additionally, we explore essential properties including their impact on mass and energy conservation. Moreover, we present and evaluate an AMR solution transfer strategy for the tree-based AMR approach that is simple to implement, memory-efficient, and ensures conservation for both flow in the box and sphere. Furthermore, we discuss scalability, performance portability, and the practical utility of the AMR methodology within an NWP framework -- crucial considerations in selecting an AMR approach. The current de facto standard for mesh refinement in NWP employs a relatively simplistic approach of static nested grids, either within a general circulation model or a separately operated regional model with loose one-way synchronization. It is our hope that this study will stimulate further interest in the adoption of AMR frameworks like AMReX in NWP. These frameworks offer a triple advantage: a robust dynamic AMR for tracking localized and consequential features such as tropical cyclones, extreme scalability, and performance portability. 

   more » « less
3. Compact binary coalescences: constraints on waveforms

   https://doi.org/10.1007/s10714-020-02764-1  

   Ashtekar, A.; De Lorenzo, T.; Khera, N. (October 2020, General relativity and gravitation)

   Gravitational waveforms for compact binary coalescences (CBCs) have been invaluable for detections by the LIGO-Virgo collaboration. They are obtained by a combination of semi-analytical models and numerical simulations. So far systematic errors arising from these procedures appear to be less than statistical ones. However, the significantly enhanced sensitivity of the new detectors that will become operational in the near future will require waveforms to be much more accurate. This task would be facilitated if one has a variety of cross-checks to evaluate accuracy, particularly in the regions of parameter space where numerical simulations are sparse. Currently errors are estimated by comparing the candidate waveforms with the numerical relativity (NR) ones, which are taken to be exact. The goal of this paper is to propose a qualitatively different tool. We show that full non-linear general relativity (GR) imposes an infinite number of sharp constraints on the CBC waveforms. These can provide clear-cut measures to evaluate the accuracy of candidate waveforms against exact GR, help find systematic errors, and also provide external checks on NR simulations themselves. 

   more » « less
4. A GPU-Accelerated AMR Solver for Gravitational Wave Propagation

   https://doi.org/10.1109/SC41404.2022.00080  

   Fernando, Milinda; Neilsen, David; Hirschmann, Eric; Zlochower, Yosef; Sundar, Hari; Ghattas, Omar; Biros, George (November 2022, SC22: International Conference for High Performance Computing, Networking, Storage and Analysis)

   Simulations to calculate a single gravitational waveform (GW) can take several weeks. Yet, thousands of such simulations are needed for the detection and interpretation of gravitational waves. Future detectors will require even more accurate waveforms than those currently used. We present here the first large scale, adaptive mesh, multi-GPU numerical relativity (NR) code together with performance analysis and benchmarking. While comparisons are difficult to make, our GPU extension of the Dendro-GR NR code achieves a 6x speedup over existing state-of-the-art codes. We achieve 800 GFlops/s on a single NVIDIA A100 GPU with an overall 2.5x speedup over a two-socket, 128-core AMD EPYC 7763 CPU node with an equivalent CPU implementation. We present detailed performance analyses, parallel scalability results, and accuracy assessments for GWs computed for mass ratios q=1,2,4. We also present strong scalability up to 8 A100s and weak scaling up to 229,376 ×86 cores on the Texas Advanced Computing Center's Frontera system. 

   more » « less
5. Structured Adaptive Mesh Refinement Adaptations to Retain Performance Portability With Increasing Heterogeneity

   https://doi.org/10.1109/MCSE.2021.3099603  

   Dubey, Anshu; Berzins, Martin; Burstedde, Carsten; Norman, Michael L.; Unat, Didem; Wahib, Mohammed (September 2021, Computing in Science & Engineering)

   Hinsen, Konrad; Dubey, Anshu (Ed.)

   Adaptive mesh refinement (AMR) is an important method that enables many mesh-based applications to run at effectively higher resolution within limited computing resources by allowing high resolution only where really needed. This advantage comes at a cost, however: greater complexity in the mesh management machinery and challenges with load distribution. With the current trend of increasing heterogeneity in hardware architecture, AMR presents an orthogonal axis of complexity. The usual techniques, such as asynchronous communication and hierarchy management for parallelism and memory that are necessary to obtain reasonable performance are very challenging to reason about with AMR. Different groups working with AMR are bringing different approaches to this challenge. Here, we examine the design choices of several AMR codes and also the degree to which demands placed on them by their users influence these choices. 

   more » « less