More Like this

1. Randomized Automatic Differentiation

   Oktay, Deniz; McGreivy, Nick; Aduol, Joshua; Beatson, Alex; Adams, Ryan P. (January 2021, International Conference on Learning Representations (ICLR))

   null (Ed.)

   The successes of deep learning, variational inference, and many other fields have been aided by specialized implementations of reverse-mode automatic differentiation (AD) to compute gradients of mega-dimensional objectives. The AD techniques underlying these tools were designed to compute exact gradients to numerical precision, but modern machine learning models are almost always trained with stochastic gradient descent. Why spend computation and memory on exact (minibatch) gradients only to use them for stochastic optimization? We develop a general framework and approach for randomized automatic differentiation (RAD), which can allow unbiased gradient estimates to be computed with reduced memory in return for variance. We examine limitations of the general approach, and argue that we must leverage problem specific structure to realize benefits. We develop RAD techniques for a variety of simple neural network architectures, and show that for a fixed memory budget, RAD converges in fewer iterations than using a small batch size for feedforward networks, and in a similar number for recurrent networks. We also show that RAD can be applied to scientific computing, and use it to develop a low-memory stochastic gradient method for optimizing the control parameters of a linear reaction-diffusion PDE representing a fission reactor. 

   more » « less
2. Orbital differentiation in Hund metals

   https://doi.org/10.1103/PhysRevB.100.115159  

   Kugler, Fabian B.; Lee, Seung-Sup B.; Weichselbaum, Andreas; Kotliar, Gabriel; von Delft, Jan (September 2019, Physical Review B)
3. Free-Radical-Mediated Glycan Isomer Differentiation

   https://doi.org/10.1021/acs.analchem.0c02213  

   Murtada, Rayan; Fabijanczuk, Kimberly; Gaspar, Kaylee; Dong, Xueming; Zeyad Alzarieni, Kawthar; Calix, Kimberly; Manriquez, Edgar; Bakestani, Rose Mery; Kenttämaa, Hilkka I.; Gao, Jinshan (September 2020, Analytical chemistry)

   The inherent structural complexity and diversity of glycans pose a major analytical challenge to their structural analysis. Radical chemistry has gained considerable momentum in the field of mass spectrometric biomolecule analysis, including proteomics, glycomics, and lipidomics. Herein, seven isomeric disaccharides and two isomeric tetrasaccharides with subtle structural differences are distinguished rapidly and accurately via one-step radical-induced dissociation. The free-radical-activated glycan-sequencing reagent (FRAGS) selectively conjugates to the unique reducing terminus of glycans in which a localized nascent free radical is generated upon collisional activation and simultaneously induces glycan fragmentation. Higher-energy collisional dissociation (HCD) and collision-induced dissociation (CID) are employed to provide complementary structural information for the identification and discrimination of glycan isomers by providing different fragmentation pathways to generate informative, structurally significant product ions. Furthermore, multiple-stage tandem mass spectrometry (MS3 CID) provides supplementary and valuable structural information through the generation of characteristic parent-structure-dependent fragment ions. 

   more » « less
4. GPU Accelerated Automatic Differentiation With Clad

   Ifrim, Ioana; Vassilev, Vassil; Lange, David (October 2022, 20th International Workshop on Advanced Computing and Analysis Techniques in Physics Research)

   Automatic Differentiation (AD) is instrumental for science and industry. It is a tool to evaluate the derivative of a function specified through a computer program. The range of AD application domain spans from Machine Learning to Robotics to High Energy Physics. Computing gradients with the help of AD is guaranteed to be more precise than the numerical alternative and have a low, constant factor more arithmetical operations compared to the original function. Moreover, AD applications to domain problems typically are computationally bound. They are often limited by the computational requirements of high-dimensional parameters and thus can benefit from parallel implementations on graphics processing units (GPUs). Clad aims to enable differential analysis for C/C++ and CUDA and is a compiler-assisted AD tool available both as a compiler extension and in ROOT. Moreover, Clad works as a plugin extending the Clang compiler; as a plugin extending the interactive interpreter Cling; and as a Jupyter kernel extension based on xeus-cling. We demonstrate the advantages of parallel gradient computations on GPUs with Clad. We explain how to bring forth a new layer of optimization and a proportional speed up by extending Clad to support CUDA. The gradients of well-behaved C++ functions can be automatically executed on a GPU. The library can be easily integrated into existing frameworks or used interactively. Furthermore, we demonstrate the achieved application performance improvements, including (~10x) in ROOT histogram fitting and corresponding performance gains from offloading to GPUs. 

   more » « less
5. Learning Set Functions with Implicit Differentiation

   Özcan, Gözde; Shi, Chengzhi; Ioannidis, Stratis (February 2025, Association for the Advancement of Artificial Intelligence (AAAI))

   A recent work introduces the problem of learning set functions from data generated by a so-called optimal subset oracle. Their approach approximates the underlying utility function with an energy-based model, whose parameters are estimated via mean-field variational inference. This approximation reduces to fixed point iterations; however, as the number of iterations increases, automatic differentiation quickly becomes computationally prohibitive due to the size of the Jacobians that are stacked during backpropagation. We address this challenge with implicit differentiation and examine the convergence conditions for the fixed-point iterations. We empirically demonstrate the efficiency of our method on synthetic and real-world subset selection applications including product recommendation, set anomaly detection and compound selection tasks. 

   more » « less