More Like this

1. Hovering Flight of Soft-Actuated Insect-Scale Micro Aerial Vehicles using Deep Reinforcement Learning

   https://doi.org/10.1109/RoboSoft63089.2025.11020855  

   Hsiao, Yi-Hsuan; Chen, Wei-Tung; Chang, Yun-Sheng; Agrawal, Pulkit; Chen, YuFeng (April 2025, IEEE)

   Soft-actuated insect-scale micro aerial vehicles (IMAVs) pose unique challenges for designing robust and computationally efficient controllers. At the millimeter scale, fast robot dynamics (∼ms), together with system delay, model uncertainty, and external disturbances significantly affect flight performances. Here, we design a deep reinforcement learning (RL) controller that addresses system delay and uncertainties. To initialize this neural network (NN) controller, we propose a modified behavior cloning (BC) approach with state-action re-matching to account for delay and domain-randomized expert demonstration to tackle uncertainty. Then we apply proximal policy optimization (PPO) to fine-tune the policy during RL, enhancing performance and smoothing commands. In simulations, our modified BC substantially increases the mean reward compared to baseline BC; and RL with PPO improves flight quality and reduces command fluctuations. We deploy this controller on two different insect-scale aerial robots that weigh 720 mg and 850 mg, respectively. The robots demonstrate multiple successful zero-shot hovering flights, with the longest lasting 50 seconds and root-mean-square errors of 1.34 cm in lateral direction and 0.05 cm in altitude, marking the first end-to-end deep RL-based flight on soft-driven IMAVs. 

   more » « less
2. Risk‐aware safe reinforcement learning for control of stochastic linear systems

   https://doi.org/10.1002/asjc.3800  

   Esmaeili, Babak; Niknejad, Nariman; Modares, Hamidreza (July 2025, Asian Journal of Control)

   Abstract This paper presents a risk‐aware safe reinforcement learning (RL) control design for stochastic discrete‐time linear systems. Rather than using a safety certifier to myopically intervene with the RL controller, a risk‐informed safe controller is also learned besides the RL controller, and the RL and safe controllers are combined together. Several advantages come along with this approach: (1) High‐confidence safety can be certified without relying on a high‐fidelity system model and using limited data available, (2) myopic interventions and convergence to an undesired equilibrium can be avoided by deciding on the contribution of two stabilizing controllers, and (3) highly efficient and computationally tractable solutions can be provided by optimizing over a scalar decision variable and linear programming polyhedral sets. To learn safe controllers with a large invariant set, piecewise affine controllers are learned instead of linear controllers. To this end, the closed‐loop system is first represented using collected data, a decision variable, and noise. The effect of the decision variable on the variance of the safe violation of the closed‐loop system is formalized. The decision variable is then designed such that the probability of safety violation for the learned closed‐loop system is minimized. It is shown that this control‐oriented approach reduces the data requirements and can also reduce the variance of safety violations. Finally, to integrate the safe and RL controllers, a new data‐driven interpolation technique is introduced. This method aims to maintain the RL agent's optimal implementation while ensuring its safety within environments characterized by noise. The study concludes with a simulation example that serves to validate the theoretical results. 

   more » « less
3. Reinforcement Learning from Delayed Observations via World Models

   Karamzade, Armin; Kim, Kyungmin; Kalsi, Montek; Fox, Roy (August 2024, CoRR abs/2403.12309)

   In standard reinforcement learning settings, agents typically assume immediate feedback about the effects of their actions after taking them. However, in practice, this assumption may not hold true due to physical constraints and can significantly impact the performance of learning algorithms. In this paper, we address observation delays in partially observable environments. We propose leveraging world models, which have shown success in integrating past observations and learning dynamics, to handle observation delays. By reducing delayed POMDPs to delayed MDPs with world models, our methods can effectively handle partial observability, where existing approaches achieve sub-optimal performance or degrade quickly as observability decreases. Experiments suggest that one of our methods can outperform a naive model-based approach by up to 250%. Moreover, we evaluate our methods on visual delayed environments, for the first time showcasing delay-aware reinforcement learning continuous control with visual observations. 

   more » « less
4. Emphasizing Electro-Thermal Dynamics for Design, Modeling, and Control of Microgrids

   https://doi.org/10.1109/ACCESS.2025.3593387  

   Ouedraogo, Asmaou S; Docimo, Donald J (July 2025, IEEE Access)

   This article studies the design, modeling, and control of microgrid systems with the inclusion of internal electro-thermal dynamics. Microgrids play a vital role in integrating renewable energy and enabling distributed energy systems. However, their complexity arising from diverse and dynamic components necessitates advanced control strategies. While existing works often utilize model-based controllers, the focus is primarily on electrical dynamics, with limited attention to the thermal behavior of components. The intricate interplay between electrical and thermal power terms heavily impacts component behavior, exemplified by the dependency of photovoltaic (PV) module electricity production on temperature. This article addresses the limited studies on electro-thermal microgrid dynamics through three contributions. First, a candidate microgrid design is developed to utilize electro-thermal knowledge, incorporating active cooling for PVs. Second, a graph-based modeling methodology is expanded to represent microgrid component- and system-level dynamics. Third, a hierarchical control framework is developed to define controllers for microgrids using the graphical model. Controllers produced from the framework enable management of electro-thermal behavior while adhering to battery charge limits. Case studies utilizing realistic environmental data are explored to evaluate the performance of the proposed system. Results indicate design and model-based control that integrate electro-thermal dynamics provide improvements to energy generation and performance even under nonideal conditions. 

   more » « less
5. RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systems

   https://doi.org/10.1145/3529375  

   Liu, Bai; Xie, Qiaomin; Modiano, Eytan (March 2022, ACM Transactions on Modeling and Performance Evaluation of Computing Systems)

   With the rapid advance of information technology, network systems have become increasingly complex and hence the underlying system dynamics are often unknown or difficult to characterize. Finding a good network control policy is of significant importance to achieve desirable network performance (e.g., high throughput or low delay). In this work, we consider using model-based reinforcement learning (RL) to learn the optimal control policy for queueing networks so that the average job delay (or equivalently the average queue backlog) is minimized. Traditional approaches in RL, however, cannot handle the unbounded state spaces of the network control problem. To overcome this difficulty, we propose a new algorithm, called RL for Queueing Networks (RL-QN), which applies model-based RL methods over a finite subset of the state space while applying a known stabilizing policy for the rest of the states. We establish that the average queue backlog under RL-QN with an appropriately constructed subset can be arbitrarily close to the optimal result. We evaluate RL-QN in dynamic server allocation, routing, and switching problems. Simulation results show that RL-QN minimizes the average queue backlog effectively. 

   more » « less