More Like this

1. Stochastic subspace approach to gradient-free optimization in high dimensions

   Kozak, David ; Becker, Stephen ; Doostan, Alireza ; Tenorio, Luis. ( January 2021 , Computational optimization and applications)

   null (Ed.)

   We present a stochastic descent algorithm for unconstrained optimization that is particularly efficient when the objective function is slow to evaluate and gradients are not easily obtained, as in some PDE-constrained optimization and machine learning problems. The algorithm maps the gradient onto a low-dimensional ran- dom subspace of dimension at each iteration, similar to coordinate descent but without restricting directional derivatives to be along the axes. Without requiring a full gradient, this mapping can be performed by computing directional deriva- tives (e.g., via forward-mode automatic differentiation). We give proofs for conver- gence in expectation under various convexity assumptions as well as probabilistic convergence results under strong-convexity. Our method provides a novel extension to the well-known Gaussian smoothing technique to descent in subspaces of dimen- sion greater than one, opening the doors to new analysis of Gaussian smoothing when more than one directional derivative is used at each iteration. We also provide a finite-dimensional variant of a special case of the Johnson–Lindenstrauss lemma. Experimentally, we show that our method compares favorably to coordinate descent, Gaussian smoothing, gradient descent and BFGS (when gradients are calculated via forward-mode automatic differentiation) on problems from the machine learning and shape optimization literature. 

   more » « less
2. High Probability Complexity Bounds for Line Search Based on Stochastic Oracles

   Billy Jin, Katya Scheinberg ( January 2021 , Advances in Neural Information Processing Systems 34 (NeurIPS 2021))

   We consider a line-search method for continuous optimization under a stochastic setting where the function values and gradients are available only through inexact probabilistic zeroth and first-order oracles. These oracles capture multiple standard settings including expected loss minimization and zeroth-order optimization. Moreover, our framework is very general and allows the function and gradient estimates to be biased. The proposed algorithm is simple to describe, easy to implement, and uses these oracles in a similar way as the standard deterministic line search uses exact function and gradient values. Under fairly general conditions on the oracles, we derive a high probability tail bound on the iteration complexity of the algorithm when applied to non-convex smooth functions. These results are stronger than those for other existing stochastic line search methods and apply in more general settings. 

   more » « less
3. High Probability Complexity Bounds for Line Search Based on Stochastic Oracles

   Jin, B. ( October 2021 , Advances in neural information processing systems)

   Ranzato, M.: ; Dauphin, Y. ; Liang, P.S. ; Wortman Vaughan, J. (Ed.)

   We consider a line-search method for continuous optimization under a stochastic setting where the function values and gradients are available only through inexact probabilistic zeroth and first-order oracles. These oracles capture multiple stan- dard settings including expected loss minimization and zeroth-order optimization. Moreover, our framework is very general and allows the function and gradient estimates to be biased. The proposed algorithm is simple to describe, easy to im- plement, and uses these oracles in a similar way as the standard deterministic line search uses exact function and gradient values. Under fairly general conditions on the oracles, we derive a high probability tail bound on the iteration complexity of the algorithm when applied to non-convex smooth functions. These results are stronger than those for other existing stochastic line search methods and apply in more general settings. 

   more » « less
4. A high-order meshless Galerkin method for semilinear parabolic equations on spheres

   https://doi.org/doi:10.1007/s00211-018-01021-7  

   Kunemund, Jens ; Narcowich, Francis J. ; Ward, Joseph D. ; Wendland, Holger ( July 2019 , Numerische Mathematik)

   We describe a novel meshless Galerkin method for numerically solving semilinear parabolic equations on spheres. The new approximation method is based upon a discretization in space using spherical basis functions in a Galerkin approximation. As our spatial approximation spaces are built with spherical basis functions, they can be of arbitrary order and do not require the construction of an underlying mesh. We will establish convergence of the meshless method by adapting, to the sphere, a convergence result due to Thom\'ee and Wahlbin. To do this requires proving new approximation results, including a novel inverse or Nikolskii inequality for spherical basis functions. We also discuss how the integrals in the Galerkin method can accurately and more efficiently be computed using a recently developed quadrature rule. These new quadrature formulas also apply to Galerkin approximations of elliptic partial differential equations on the sphere. Finally, we provide several numerical examples, including the Allen-Cahn for the sphere. 

   more » « less
5. SUCAG: Stochastic Unbiased Curvature-aided Gradient Method for Distributed Optimization

   https://doi.org/10.1109/CDC.2018.8619336  

   Wai, Hoi-To ; Freris, Nikolaos M. ; Nedic, Angelia ; Scaglione, Anna ( December 2018 , 2018 IEEE Conference on Decision and Control (CDC))

   We propose and analyze a new stochastic gradient method, which we call Stochastic Unbiased Curvature-aided Gradient (SUCAG), for finite sum optimization problems. SUCAG constitutes an unbiased total gradient tracking technique that uses Hessian information to accelerate convergence. We analyze our method under the general asynchronous model of computation, in which each function is selected infinitely often with possibly unbounded (but sublinear) delay. For strongly convex problems, we establish linear convergence for the SUCAG method. When the initialization point is sufficiently close to the optimal solution, the established convergence rate is only dependent on the condition number of the problem, making it strictly faster than the known rate for the SAGA method. Furthermore, we describe a Markov-driven approach of implementing the SUCAG method in a distributed asynchronous multi-agent setting, via gossiping along a random walk on an undirected communication graph. We show that our analysis applies as long as the graph is connected and, notably, establishes an asymptotic linear convergence rate that is robust to the graph topology. Numerical results demonstrate the merits of our algorithm over existing methods. 

   more » « less