More Like this

1. Conversion of a Class of Stochastic Control Problems to Fundamental-Solution Deterministic Control Problems

   McEneaney, William M; Dower, Peter M. (July 2020, Proceedings of the American Control Conference)

   A new optimal control based representation for stationary action trajectories is constructed by exploiting connections between semiconvexity, semiconcavity, and stationarity. This new representation is used to verify a known two-point boundary value problem characterization of stationary action. 

   more » « less
2. Non-stationary Reinforcement Learning under General Function Approximation

   Feng, Songtao and; Yin, Ming; Huang, Ruiquan; Wang, Yu-Xiang; Yang, Jing; Liang, Yingbin (July 2023, Proceedings of Machine Learning Research)

   General function approximation is a powerful tool to handle large state and action spaces in a broad range of reinforcement learning (RL) scenarios. However, theoretical understanding of non-stationary MDPs with general function approximation is still limited. In this paper, we make the first such an attempt. We first propose a new complexity metric called dynamic Bellman Eluder (DBE) dimension for non-stationary MDPs, which subsumes majority of existing tractable RL problems in static MDPs as well as non-stationary MDPs. Based on the proposed complexity metric, we propose a novel confidence-set based model-free algorithm called SW-OPEA, which features a sliding window mechanism and a new confidence set design for non-stationary MDPs. We then establish an upper bound on the dynamic regret for the proposed algorithm, and show that SW-OPEA is provably efficient as long as the variation budget is not significantly large. We further demonstrate via examples of non-stationary linear and tabular MDPs that our algorithm performs better in small variation budget scenario than the existing UCB-type algorithms. To the best of our knowledge, this is the first dynamic regret analysis in non-stationary MDPs with general function approximation. 

   more » « less
3. Sampling Networks and Aggregate Simulation for Online POMDP Planning

   Cui, H; Khardon, R. (December 2019, Advances in neural information processing systems)

   The paper introduces a new algorithm for planning in partially observable Markov decision processes (POMDP) based on the idea of aggregate simulation. The algorithm uses product distributions to approximate the belief state and shows how to build a representation graph of an approximate action-value function over belief space. The graph captures the result of simulating the model in aggregate under independence assumptions, giving a symbolic representation of the value function. The algorithm supports large observation spaces using sampling networks, a representation of the process of sampling values of observations, which is integrated into the graph representation. Following previous work in MDPs this approach enables action selection in POMDPs through gradient optimization over the graph representation. This approach complements recent algorithms for POMDPs which are based on particle representations of belief states and an explicit search for action selection. Our approach enables scaling to large factored action spaces in addition to large state spaces and observation spaces. An experimental evaluation demonstrates that the algorithm provides excellent performance relative to state of the art in large POMDP problems. 

   more » « less
4. Kink propagation in the Artificial Axon

   https://doi.org/10.1209/0295-5075/ac44e2  

   Qi, Xinyi; Zocchi, Giovanni (January 2022, Europhysics Letters)

   Abstract The Artificial Axon is a unique synthetic system, based on biomolecular components, which supports action potentials. Here we consider, theoretically, the corresponding space extended system, and discuss the occurrence of solitary waves, or kinks. Such structures are indeed observed in living systems. In contrast to action potentials, stationary kinks are possible. We point out an analogy with the interface separating two condensed matter phases, though our kinks are always non-equilibrium, dissipative structures, even when stationary. 

   more » « less
5. Generalized Clebsch Variables for Compressible Ideal Fluids: Initial Conditions and Approximations of the Hamiltonian

   https://doi.org/10.3390/fluids7040122  

   Rumpf, Benno; Lvov, Yuri V. (April 2022, Fluids)

   Clebsch variables provide a canonical representation of ideal flows that is, in practice, difficult to handle: while the velocity field is a function of the Clebsch variables and their gradients, constructing the Clebsch variables from the velocity field is not trivial. We introduce an extended set of Clebsch variables that circumvents this problem. We apply this method to a compressible, chemically inhomogeneous, and rotating ideal fluid in a gravity field. A second difficulty, the secular growth of canonical variables even for stationary states of stratified fluids, makes expansions of the Hamiltonian in Clebsch variables problematic. We give a canonical transformation that associates a stationary state of the canonical variables with the stationary state of the fluid; the new set of variables permits canonical approximations of the dynamics. We apply this to a compressible stratified ideal fluid with the aim to facilitate forthcoming studies of wave turbulence of internal waves. 

   more » « less