More Like this

1. Stochastic control and non-equilibrium thermodynamics: fundamental limits

   https://doi.org/10.1109/TAC.2019.2939625  

   Chen, Yongxin; Georgiou, Tryphon; Tannenbaum, Allen (January 2020, IEEE Transactions on Automatic Control)

   We consider damped stochastic systems in a controlled (time-varying) potential and study their transition between specified Gibbs-equilibria states in finite time. By the second law of thermody- namics, the minimum amount of work needed to transition from one equilibrium state to another is the difference between the Helmholtz free energy of the two states and can only be achieved by a reversible (infinitely slow) process. The minimal gap between the work needed in a finite-time transition and the work during a reversible one, turns out to equal the square of the optimal mass transport (Wasserstein- 2) distance between the two end-point distributions times the inverse of the duration needed for the transition. This result, in fact, relates non-equilibrium optimal control strategies (protocols) to gradient flows of entropy functionals via the Jordan-Kinderlehrer-Otto scheme. The purpose of this paper is to introduce ideas and results from the emerging field of stochastic thermodynamics in the setting of classical regulator theory, and to draw connections and derive such fundamental relations from a control perspective in a multivariable setting. 

   more » « less
2. Learning-Based Adaptive Optimal Control of Linear Time-Delay Systems: A Policy Iteration Approach

   https://doi.org/10.1109/TAC.2023.3273786  

   Cui, Leilei; Pang, Bo; Jiang, Zhong-Ping (May 2023, IEEE Transactions on Automatic Control)

   This paper studies the adaptive optimal control problem for a class of linear time-delay systems described by delay differential equations (DDEs). A crucial strategy is to take advantage of recent developments in reinforcement learning (RL) and adaptive dynamic programming (ADP) and develop novel methods to learn adaptive optimal controllers from finite samples of input and state data. In this paper, the data-driven policy iteration (PI) is proposed to solve the infinite-dimensional algebraic Riccati equation (ARE) iteratively in the absence of exact model knowledge. Interestingly, the proposed recursive PI algorithm is new in the present context of continuous-time time-delay systems, even when the model knowledge is assumed known. The efficacy of the proposed learning-based control methods is validated by means of practical applications arising from metal cutting and autonomous driving. 

   more » « less
3. Optimal finite-time processes in weakly driven overdamped Brownian motion

   https://doi.org/10.1088/2399-6528/ac871d  

   Nazé, Pierre; Deffner, Sebastian; Bonança, Marcus V. S. (August 2022, Journal of Physics Communications)

   Abstract The complete physical understanding of the optimization of the thermodynamic work still is an important open problem in stochastic thermodynamics. We address this issue using the Hamiltonian approach of linear response theory in finite time and weak processes. We derive the Euler–Lagrange equation associated and discuss its main features, illustrating them using the paradigmatic example of driven Brownian motion in overdamped regime. We show that the optimal protocols obtained either coincide, in the appropriate limit, with the exact solutions by stochastic thermodynamics or can be even identical to them, presenting the well-known jumps. However, our approach reveals that jumps at the extremities of the process are a good optimization strategy in the regime of fast but weak processes for any driven system. Additionally, we show that fast-but-weak optimal protocols are time-reversal symmetric, a property that has until now remained hidden in the exact solutions far from equilibrium. 

   more » « less
4. A Neural Network Approach for Stochastic Optimal Control

   Li, Xingjian; Verma, Deepanshu; Ruthotto, Lars (June 2024, SIAM journal on scientific computing)

   We present a neural network approach for approximating the value function of high- dimensional stochastic control problems. Our training process simultaneously updates our value function estimate and identifies the part of the state space likely to be visited by optimal trajectories. Our approach leverages insights from optimal control theory and the fundamental relation between semi-linear parabolic partial differential equations and forward-backward stochastic differential equations. To focus the sampling on relevant states during neural network training, we use the stochastic Pontryagin maximum principle (PMP) to obtain the optimal controls for the current value function estimate. By design, our approach coincides with the method of characteristics for the non-viscous Hamilton-Jacobi-Bellman equation arising in deterministic control problems. Our training loss consists of a weighted sum of the objective functional of the control problem and penalty terms that enforce the HJB equations along the sampled trajectories. Importantly, training is unsupervised in that it does not require solutions of the control problem. Our numerical experiments highlight our scheme’s ability to identify the relevant parts of the state space and produce meaningful value estimates. Using a two-dimensional model problem, we demonstrate the importance of the stochastic PMP to inform the sampling and compare to a finite element approach. With a nonlinear control affine quadcopter example, we illustrate that our approach can handle complicated dynamics. For a 100-dimensional benchmark problem, we demonstrate that our approach improves accuracy and time-to-solution and, via a modification, we show the wider applicability of our scheme. 

   more » « less
5. Linear quadratic stochastic optimal control problems with operator coefficients: open-loop solutions

   https://doi.org/10.1051/cocv/2018013  

   Wei, Qingmeng; Yong, Jiongmin; Yu, Zhiyong (January 2019, ESAIM: Control, Optimisation and Calculus of Variations)

   An optimal control problem is considered for linear stochastic differential equations with quadratic cost functional. The coefficients of the state equation and the weights in the cost functional are bounded operators on the spaces of square integrable random variables. The main motivation of our study is linear quadratic (LQ, for short) optimal control problems for mean-field stochastic differential equations. Open-loop solvability of the problem is characterized as the solvability of a system of linear coupled forward-backward stochastic differential equations (FBSDE, for short) with operator coefficients, together with a convexity condition for the cost functional. Under proper conditions, the well-posedness of such an FBSDE, which leads to the existence of an open-loop optimal control, is established. Finally, as applications of our main results, a general mean-field LQ control problem and a concrete mean-variance portfolio selection problem in the open-loop case are solved. 

   more » « less