More Like this

1. DJ4Earth: Differentiable, and Performance‐Portable Earth System Modeling via Program Transformations

   https://doi.org/10.1029/2025MS005615  

   Moses, William S; Cheng, Gong; Churavy, Valentin; Gelbrecht, Maximilian; Klöwer, Milan; Kump, Joseph; Morlighem, Mathieu; Williamson, Sarah; Apte, Dhruv; Berg, Paul; et al (May 2026, Journal of Advances in Modeling Earth Systems)

   Abstract Differentiable Earth system models (ESMs) enable powerful applications such as sensitivity analysis, gradient‐based calibration, state estimation, boundary flux inversions, uncertainty quantification, and online machine learning. Reverse‐mode automatic differentiation (AD) efficiently provides gradients for such tasks, yet models have rarely included this capability because of complex, bespoke numerical algorithms. As part of theDifferentiable programming in Julia for Earth system modeling (DJ4Earth)initiative, we present improved capabilities of the AD tool Enzyme.jl and the new compiler transpilation tool Reactant.jl, augmented by sophisticated checkpointing algorithms, which, together make general‐purpose AD tractable and efficient for full‐fledged ESM components written in Julia. Operating at the low‐level virtual machine intermediate representation or multi‐level intermediate representation compiler levels, these frameworks support mutable memory, custom kernels, and compiler optimizations before and after differentiation. Julia‐specific challenges related to just‐in‐time compilation and garbage collection are handled efficiently. Reactant further enables automatic performance portability across central processing units, graphics processing units, and tensor processing units, facilitating use of emerging AI‐customized high‐performance computing architectures. We demonstrate these frameworks on four Julia‐based ESM components featuring diverse spatial discretizations and numerical algorithms: the rotating‐sphere shallow water model ShallowWaters.jl, the finite‐volume ocean model Oceananigans.jl, the finite‐element ice sheet model DJUICE.jl, and the spectral atmospheric model SpeedyWeather.jl. Across these ESM components, our tools compute efficient and correct gradients. These results establish a foundation for differentiable, high‐performance and performance‐portable ESMs that can integrate neural networks for unresolved processes, trained online, enabling next‐generation hybrid physics–machine learning ESMs constrained by physical dynamics and observations. 

   more » « less
2. Instead of Rewriting Foreign Code for Machine Learning, Automatically Synthesize Fast Gradients

   Moses, William; Churavy, Valentin (January 2020, Advances in neural information processing systems)

   Applying differentiable programming techniques and machine learning algorithms to foreign programs requires developers to either rewrite their code in a machine learning framework, or otherwise provide derivatives of the foreign code. This paper presents Enzyme, a high-performance automatic differentiation (AD) compiler plugin for the LLVM compiler framework capable of synthesizing gradients of statically analyzable programs expressed in the LLVM intermediate representation (IR). Enzyme synthesizes gradients for programs written in any language whose compiler targets LLVM IR including C, C++, Fortran, Julia, Rust, Swift, MLIR, etc., thereby providing native AD capabilities in these languages. Unlike traditional source-to-source and operator-overloading tools, Enzyme performs AD on optimized IR. On a machine-learning focused benchmark suite including Microsoft's ADBench, AD on optimized IR achieves a geometric mean speedup of 4.2 times over AD on IR before optimization allowing Enzyme to achieve state-of-the-art performance. Packaging Enzyme for PyTorch and TensorFlow provides convenient access to gradients of foreign code with state-of-the-art performance, enabling foreign code to be directly incorporated into existing machine learning workflows. 

   more » « less
3. Scalable Automatic Differentiation of Multiple Parallel Paradigms through Compiler Augmentation

   https://doi.org/10.1109/SC41404.2022.00065  

   Moses, William S; Narayanan, Sri Hari; Paehler, Ludger; Churavy, Valentin; Schanen, Michel; Hückelheim, Jan; Doerfert, Johannes; Hovland, Paul (November 2022, International Conference for High Performance Computing Networking Storage and Analysis)

   Derivatives are key to numerous science, engineering, and machine learning applications. While existing tools generate derivatives of programs in a single language, modern parallel applications combine a set of frameworks and languages to leverage available performance and function in an evolving hardware landscape. We propose a scheme for differentiating arbitrary DAG-based parallelism that preserves scalability and efficiency, implemented into the LLVM-based Enzyme automatic differentiation framework. By integrating with a full-fledged compiler backend, Enzyme can differentiate numerous parallel frameworks and directly control code generation. Combined with its ability to differentiate any LLVM-based language, this flexibility permits Enzyme to leverage the compiler tool chain for parallel and differentiation-specific optimizations. We differentiate nine distinct versions of the LULESH and miniBUDE applications, written in different programming languages (C++, Julia) and parallel frameworks (OpenMP, MPI, RAJA, Julia tasks, MPI.jl), demonstrating similar scalability to the original program. On benchmarks with 64 threads or nodes, we find a differentiation overhead of 3.4 - 6.8× on C++ and 5.4 - 12.5× on Julia. 

   more » « less
4. Reverse-mode automatic differentiation and optimization of GPU kernels via enzyme

   https://doi.org/10.1145/3458817.3476165  

   Moses, William S.; Churavy, Valentin; Paehler, Ludger; Hückelheim, Jan; Narayanan, Sri Hari; Schanen, Michel; Doerfert, Johannes (November 2021, SC '21: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis)

   Computing derivatives is key to many algorithms in scientific computing and machine learning such as optimization, uncertainty quantification, and stability analysis. Enzyme is a LLVM compiler plugin that performs reverse-mode automatic differentiation (AD) and thus generates high performance gradients of programs in languages including C/C++, Fortran, Julia, and Rust. Prior to this work, Enzyme and other AD tools were not capable of generating gradients of GPU kernels. Our paper presents a combination of novel techniques that make Enzyme the first fully automatic reversemode AD tool to generate gradients of GPU kernels. Since unlike other tools Enzyme performs automatic differentiation within a general-purpose compiler, we are able to introduce several novel GPU and AD-specific optimizations. To show the generality and efficiency of our approach, we compute gradients of five GPU-based HPC applications, executed on NVIDIA and AMD GPUs. All benchmarks run within an order of magnitude of the original program's execution time. Without GPU and AD-specific optimizations, gradients of GPU kernels either fail to run from a lack of resources or have infeasible overhead. Finally, we demonstrate that increasing the problem size by either increasing the number of threads or increasing the work per thread, does not substantially impact the overhead from differentiation. 

   more » « less
5. What questions do developers ask about Julia?

   https://doi.org/10.1145/3476883.3520205  

   Bose, Dibyendu Brinto; Gannod, Gerald C.; Rahman, Akond; Cottrell, Kaitlyn (April 2022, 2022 ACM Southeast Conference)

   The programming language Julia is designed to solve the 'two language problem', where developers who write scientific software can achieve desired performance, without sacrificing productivity. Since its inception in 2012, developers who have been using other programming languages have transitioned to Julia. A systematic investigation of the questions that developers ask about Julia can help in understanding the challenges that developers face while using Julia. Such understanding can be helpful (i) for toolsmiths who can construct tools so that developers can maximize their experience of using Julia, and (ii) for Julia language maintainers with empirical evidence on areas to improve the language as well as the Julia ecosystem. We conduct an empirical study with 3,093 Stack Overflow posts where we identify 13 categories of questions related to Julia-based software development. We observe developers to ask about a diverse set of topics, such as GC, Julia's garbage collector, JuMP, a domain-specific language constructed using Julia, and symbols, a metaprogramming utility in Julia. Based on our emerging results, we recommend enhancing support for developers with Julia-based tools and techniques for cross language transfer, type-related assistance, and package resolution. 

   more » « less