More Like this

1. A modified limited memory steepest descent method motivated by an inexact super-linear convergence rate analysis

   https://doi.org/10.1093/imanum/drz059  

   Gu, Ran; Du, Qiang (April 2020, IMA Journal of Numerical Analysis)

   null (Ed.)

   Abstract How to choose the step size of gradient descent method has been a popular subject of research. In this paper we propose a modified limited memory steepest descent method (MLMSD). In each iteration we propose a selection rule to pick a unique step size from a candidate set, which is calculated by Fletcher’s limited memory steepest descent method (LMSD), instead of going through all the step sizes in a sweep, as in Fletcher’s original LMSD algorithm. MLMSD is motivated by an inexact super-linear convergence rate analysis. The R-linear convergence of MLMSD is proved for a strictly convex quadratic minimization problem. Numerical tests are presented to show that our algorithm is efficient and robust. 

   more » « less
2. Benchmark Computation of Morphological Complexity in the Functionalized Cahn-Hilliard Gradient Flow

   https://doi.org/10.4208/cicp.OA-2023-0190  

   Andrew_Christlieb, Andrew Christlieb; Keith_Promislow, Keith Promislow; Zengqiang_Tan, Zengqiang Tan; Sulin_Wang, Sulin Wang; Brian_Wetton, Brian Wetton; Steven_M_Wise, Steven_M Wise (January 2025, Communications in Computational Physics)

   Reductions of the self-consistent mean field theory model of amphiphilic molecules in solvent can lead to a singular family of functionalized Cahn-Hilliard (FCH) energies. We modify these energies, mollifying the singularities to stabilize the computation of the gradient flows and develop a series of benchmark problems that emulate the “morphological complexity” observed in experiments. These benchmarks investigate the delicate balance between the rate of absorption of amphiphilic material onto an interface and a least energy mechanism to disperse the arriving mass. The re- sult is a trichotomy of responses in which two-dimensional interfaces either lengthen by a regularized motion against curvature, undergo pearling bifurcations, or split di- rectly into networks of interfaces. We evaluate a number of schemes that use second or- der backward differentiation formula (BDF2) type time stepping coupled with Fourier pseudo-spectral spatial discretization. The BDF2-type schemes are either based on a fully implicit time discretization with a preconditioned steepest descent (PSD) nonlin- ear solver or upon linearly implicit time discretization based on the standard implicit- explicit (IMEX) and the scalar auxiliary variable (SAV) approaches. We add an ex- ponential time differencing (ETD) scheme for comparison purposes. 

   more » « less
3. Nonlinear steepest descent on a torus: a case study of the Landau–Lifshitz equation

   https://doi.org/10.1088/1361-6544/adbe22  

   Desiraju, Harini; Its, Alexander; Prokhorov, Andrei (March 2025, Nonlinearity)

   Abstract We obtain rigorous large time asymptotics for the Landau–Lifshitz (LL) equation in the soliton free case by extending the nonlinear steepest descent method to genus 1 surfaces. The methods presented in this paper pave the way to a rigorous analysis of other integrable equations on the torus and enable asymptotic analysis on different regimes of the LL equation. 

   more » « less
4. A High Probability Analysis of Adaptive SGD with Momentum

   Li, Xiaoyu; Orabona, Francesco (January 2020, Workshop on Beyond First Order Methods in ML Systems at ICML'20)

   null (Ed.)

   Stochastic Gradient Descent (SGD) and its variants are the most used algorithms in machine learning applications. In particular, SGD with adaptive learning rates and momentum is the industry standard to train deep networks. Despite the enormous success of these methods, our theoretical understanding of these variants in the non-convex setting is not complete, with most of the results only proving convergence in expectation and with strong assumptions on the stochastic gradients. In this paper, we present a high probability analysis for adaptive and momentum algorithms, under weak assumptions on the function, stochastic gradients, and learning rates. We use it to prove for the first time the convergence of the gradients to zero in high probability in the smooth nonconvex setting for Delayed AdaGrad with momentum. 

   more » « less
5. Gradient Descent in the Absence of Global Lipschitz Continuity of the Gradients

   https://doi.org/10.1137/22M1527210  

   Patel, Vivak; Berahas, Albert S (September 2024, SIAM Journal on Mathematics of Data Science)

   Gradient descent (GD) is a collection of continuous optimization methods that have achieved immeasurable success in practice. Owing to data science applications, GD with diminishing step sizes has become a prominent variant. While this variant of GD has been well studied in the literature for objectives with globally Lipschitz continuous gradients or by requiring bounded iterates, objectives from data science problems do not satisfy such assumptions. Thus, in this work, we provide a novel global convergence analysis of GD with diminishing step sizes for differentiable nonconvex functions whose gradients are only locally Lipschitz continuous. Through our analysis, we generalize what is known about gradient descent with diminishing step sizes, including interesting topological facts, and we elucidate the varied behaviors that can occur in the previously overlooked divergence regime. Thus, we provide a general global convergence analysis of GD with diminishing step sizes under realistic conditions for data science problems. 

   more » « less