More Like this

1. Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence

   Han, Yinbin; Razaviyayn, Meisam; Xu, Renyuan (May 2025, International Conference on Machine Learning (ICML) 2025)

   Diffusion models have emerged as powerful tools for generative modeling, demonstrating exceptional capability in capturing target data distributions from large datasets. However, fine-tuning these massive models for specific downstream tasks, constraints, and human preferences remains a critical challenge. While recent advances have leveraged reinforcement learning algorithms to tackle this problem, much of the progress has been empirical, with limited theoretical understanding. To bridge this gap, we propose a stochastic control framework for fine-tuning diffusion models. Building on denoising diffusion probabilistic models as the pre-trained reference dynamics, our approach integrates linear dynamics control with Kullback–Leibler regularization. We establish the well-posedness and regularity of the stochastic control problem and develop a policy iteration algorithm (PI-FT) for numerical solution. We show that PI-FT achieves global convergence at a linear rate. Unlike existing work that assumes regularities throughout training, we prove that the control and value sequences generated by the algorithm preserve the desired regularity. Finally, we extend our framework to parametric settings for efficient implementation and demonstrate the practical effectiveness of the proposed PI-FT algorithm through numerical experiments. 

   more » « less
2. Optimal Consumption and Investment with Independent Stochastic Labor Income

   https://doi.org/10.1287/moor.2023.0119  

   Bensoussan, Alain; Park, Seyoung (February 2025, Mathematics of Operations Research)

   We develop a new dynamic continuous-time model of optimal consumption and investment to include independent stochastic labor income. We reduce the problem of solving the Bellman equation to a problem of solving an integral equation. We then explicitly characterize the optimal consumption and investment strategy as a function of income-to-wealth ratio. We provide some analytical comparative statics associated with the value function and optimal strategies. We also develop a quite general numerical algorithm for control iteration and solve the Bellman equation as a sequence of solutions to ordinary differential equations. This numerical algorithm can be readily applied to many other optimal consumption and investment problems especially with extra nondiversifiable Brownian risks, resulting in nonlinear Bellman equations. Finally, our numerical analysis illustrates how the presence of stochastic labor income affects the optimal consumption and investment strategy. Funding: A. Bensoussan was supported by the National Science Foundation under grant [DMS-2204795]. S. Park was supported by the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea, South Korea [NRF-2022S1A3A2A02089950]. 

   more » « less
3. Equilibrium strategies for time-inconsistent stochastic switching systems

   https://doi.org/10.1051/cocv/2018051  

   Mei, Hongwei; Yong, Jiongmin (January 2019, ESAIM: Control, Optimisation and Calculus of Variations)

   An optimal control problem is considered for a stochastic differential equation containing a state-dependent regime switching, with a recursive cost functional. Due to the non-exponential discounting in the cost functional, the problem is time-inconsistent in general. Therefore, instead of finding a global optimal control (which is not possible), we look for a time-consistent (approximately) locally optimal equilibrium strategy. Such a strategy can be represented through the solution to a system of partial differential equations, called an equilibrium Hamilton–Jacob–Bellman (HJB) equation which is constructed via a sequence of multi-person differential games. A verification theorem is proved and, under proper conditions, the well-posedness of the equilibrium HJB equation is established as well. 

   more » « less
4. Distributed algorithms of stochastic games for robot systems in smart manufacturing

   https://doi.org/10.1063/5.0236086  

   He, Xiongnan; Lin, Zongli; Chang, Qing (February 2025, Chaos: An Interdisciplinary Journal of Nonlinear Science)

   In this paper, we study the problem of distributed generalized stochastic Nash equilibrium seeking for robot systems over a connected undirected graph. We use the cost functions containing uncertainty to represent the system’s performance under varying conditions. To mitigate the challenges posed by this uncertainty, we employ the Tikhonov regularization scheme, which also helps us to relax the strongly monotone condition of the cost functions to the strictly monotone condition. We also consider the inequality constraints, which represent the feasible working space of robots. Additionally, auxiliary parameters are introduced in the control laws to facilitate seeing the variational generalized stochastic Nash equilibrium. The convergence of the proposed control laws is analyzed by using the operator splitting method. Finally, we demonstrate the effectiveness of the proposed algorithm through an example involving multiple robots connected through a communication network. 

   more » « less
5. Energy-based continuous inverse optimal control.

   https://doi.org/10.1109/TNNLS.2022.3168795  

   Xu, Y.; Xie, J.; Zhao, T.; Baker, C.; Zhao, Y.; Wu, Y. N. (January 2022, IEEE transactions on neural networks and learning systems)

   The problem of continuous inverse optimal control (over finite time horizon) is to learn the unknown cost function over the sequence of continuous control variables from expert demonstrations. In this article, we study this fundamental problem in the framework of energy-based model, where the observed expert trajectories are assumed to be random samples from a probability density function defined as the exponential of the negative cost function up to a normalizing constant. The parameters of the cost function are learned by maximum likelihood via an “analysis by synthesis” scheme, which iterates (1) synthesis step: sample the synthesized trajectories from the current probability density using the Langevin dynamics via back-propagation through time, and (2) analysis step: update the model parameters based on the statistical difference between the synthesized trajectories and the observed trajectories. Given the fact that an efficient optimization algorithm is usually available for an optimal control problem, we also consider a convenient approximation of the above learning method, where we replace the sampling in the synthesis step by optimization. Moreover, to make the sampling or optimization more efficient, we propose to train the energy-based model simultaneously with a top-down trajectory generator via cooperative learning, where the trajectory generator is used to fast initialize the synthesis step of the energy-based model. We demonstrate the proposed methods on autonomous driving tasks, and show that they can learn suitable cost functions for optimal control. 

   more » « less