More Like this

1. Continuous and Discrete-time Accelerated Stochastic Mirror Descent for Strongly Convex Functions

   Xu, Pan; Wang, Tianhao; Gu, Quanquan (July 2018, International Conference on Machine Learning)

   We provide a second-order stochastic differential equation (SDE), which characterizes the continuous-time dynamics of accelerated stochastic mirror descent (ASMD) for strongly convex functions. This SDE plays a central role in designing new discrete-time ASMD algorithms via numerical discretization, and providing neat analyses of their convergence rates based on Lyapunov functions. Our results suggest that the only existing ASMD algorithm, namely, AC-SA proposed in Ghadimi & Lan (2012) is one instance of its kind, and we can actually derive new instances of ASMD with fewer tuning parameters. This sheds light on revisiting accelerated stochastic optimization through the lens of SDEs, which can lead to a better understanding of acceleration in stochastic optimization, as well as new simpler algorithms. Numerical experiments on both synthetic and real data support our theory. 

   more » « less
2. Continuous-Time Reward Machines

   https://doi.org/10.24963/ijcai.2025/563  

   Falah, Amin; Guha, Shibashis; Trivedi, Ashutosh (September 2025, International Joint Conferences on Artificial Intelligence Organization)

   Reinforcement Learning (RL) is a sampling-based method for sequential decision-making, in which a learning agent iteratively converges toward an optimal policy by leveraging feedback from the environment in the form of scalar reward signals.While timing information is often abstracted in discrete-time domains, time-critical learning applications—such as queuing systems, population processes, and manufacturing systems—are naturally modeled as Continuous-Time Markov Decision Processes (CTMDPs).Since the seminal work of Bradtke and Duff, model-free RL for CTMDPs has become well-understood. However, in many practical applications, practitioners possess high-quality information about system rates derived from traditional queuing theory, which learning agents could potentially exploit to accelerate convergence. Despite this, classical RL algorithms for CTMDPs typically re-learn these parameters through sampling.In this work, we propose continuous-time reward machines (CTRMs), a novel framework that embeds reward functions and real-time state-action dynamics into a unified structure. CTRMs enable RL agents to effectively navigate dense-time environments while leveraging reward shaping and counterfactual experiences for accelerated learning. Our empirical results demonstrate CTRMs' ability to improve learning efficiency in time-critical environments. 

   more » « less
3. Inference of Utilities and Time Preference in Sequential Decision-Making

   https://doi.org/10.1007/s00245-025-10318-7  

   Cao, Haoyang; Wu, Zhengqi; Xu, Renyuan (December 2025, Applied Mathematics & Optimization)

   This paper introduces a novel stochastic control framework to enhance the capabilities of automated investment managers, or robo-advisors, by accurately inferring clients’ investment preferences from past activities. Our approach leverages a continuous-time model that incorporates utility functions and a generic discounting scheme with a time-varying rate, which can be tailored to each client’s risk tolerance, valuation of daily consumption, and significant life goals; this general discounting scheme is also referred to as the time preference. Through state augmentation, the new stochastic control framework allows for the joint inference of utilities and time preferences. We establish the corresponding dynamic programming principle and the verification theorem. Additionally, we provide sufficient conditions for the identifiability of client investment preferences. To complement our theoretical developments, we propose a learning algorithm based on maximum likelihood estimation within a discrete-time Markov Decision Process framework, augmented with entropy regularization. We prove that the Hessian matrix of the log-likelihood function is negative semi-definite at the client’s investment parameters, facilitating fast convergence of our proposed algorithm. Practical effectiveness and efficiency are showcased through two numerical examples, including Merton’s problem and an investment problem with unhedgeable risks. Our proposed framework not only advances financial technology by improving personalized investment advice but also contributes broadly to other fields such as healthcare, economics, and artificial intelligence, where understanding individual preferences is crucial. 

   more » « less
4. On Global and Local Convergence of Iterative Linear Quadratic Optimization Algorithms for Discrete Time Nonlinear Control

   Roulet, V; Srinivasa, S; Fazel, M; Harchaoui, Z (May 2025, Journal of machine learning research)

   Jaggi, Martin (Ed.)

   A classical approach for solving discrete time nonlinear control on a nite horizon consists in repeatedly minimizing linear quadratic approximations of the original problem around current candidate solutions. While widely popular in many domains, such an approach has mainly been analyzed locally. We provide detailed convergence guarantees to stationary points as well as local linear convergence rates for the Iterative Linear Quadratic Regulator (ILQR) algorithm and its Di erential Dynamic Programming (DDP) variant. For problems without costs on control variables, we observe that global convergence to minima can be ensured provided that the linearized discrete time dynamics are surjective, costs on the state variables are gradient dominated. We further detail quadratic local convergence when the costs are self-concordant. We show that surjectivity of the linearized dynamics hold for appropriate discretization schemes given the existence of a feedback linearization scheme. We present complexity bounds of algorithms based on linear quadratic approximations through the lens of generalized Gauss-Newton methods. Our analysis uncovers several convergence phases for regularized generalized Gauss-Newton algorithms. 

   more » « less
5. Higher order time discretization for the stochastic semilinear wave equation with multiplicative noise

   https://doi.org/10.1093/imanum/drad024  

   Feng, Xiaobing; Ashirbad Panda, Akash; Prohl, Andreas (May 2023, IMA Journal of Numerical Analysis)

   Abstract In this paper, a higher order time-discretization scheme is proposed, where the iterates approximate the solution of the stochastic semilinear wave equation driven by multiplicative noise with general drift and diffusion. We employ variational method for its error analysis and prove an improved convergence order of $$\frac 32$$ for the approximates of the solution. The core of the analysis is Hölder continuity in time and moment bounds for the solutions of the continuous and the discrete problem. Computational experiments are also presented. 

   more » « less