More Like this

1. Control Regularization for Reduced Variance Reinforcement Learning

   Cheng, Richard Cheng; Verma, Abhinav; Orosz, Gabor; Chaudhuri, Swarat; Yue, Yisong; Burdick, Joel (January 2019, Proceedings of Machine Learning Research)

   Dealing with high variance is a significant challenge in model-free reinforcement learning (RL). Existing methods are unreliable, exhibiting high variance in performance from run to run using different initializations/seeds. Focusing on problems arising in continuous control, we propose a functional regularization approach to augmenting model-free RL. In particular, we regularize the behavior of the deep policy to be similar to a control prior, i.e., we regularize in function space. We show that functional regularization yields a bias-variance trade-off, and propose an adaptive tuning strategy to optimize this trade-off. When the prior policy has control-theoretic stability guarantees, we further show that this regularization approximately preserves those stability guarantees throughout learning. We validate our approach empirically on a wide range of settings, and demonstrate significantly reduced variance, guaranteed dynamic stability, and more efficient learning than deep RL alone. 

   more » « less
2. QoS-Aware State-Augmented Learning for Wireless Coexistence Parameter Management

   https://doi.org/10.1109/TMLCN.2026.3707514  

   Fasihi, Mohammad Reza; Mark, Brian L; Alotaibi, Omar A (June 2026, IEEE Transactions on Machine Learning in Communications and Networking)

   Efficient and fair coexistence in unlicensed spectrum is critical for next-generation wireless networks such as 5G New Radio in Unlicensed Spectrum (NR-U) and Wi-Fi, which contend for shared resources under diverse Quality of Service (QoS) requirements. Existing reinforcement learning (RL) approaches either rely on multi-objective formulations with heuristic weight tuning or primal-dual constrained methods that struggle with stability in dynamic environments. To address these limitations, we propose QaSAL-CPM, a novel QoS-aware State-Augmented Learning framework for Coexistence Parameter Management (CPM). Building on the recently introduced concept of state augmentation, QaSAL-CPM embeds dual variables directly into the agent’s observation space, enabling real-time responsiveness to constraint violations without retraining or complex penalty tuning. This design achieves strict QoS guarantees for high-priority traffic while maintaining fairness across traffic classes, even under heavy contention. Unlike prior methods, QaSAL-CPM separates offline learning from lightweight online execution, making it practical for real-world deployments such as Ultra-Reliable Low-Latency Communications (URLLC) and the Internet of Things (IoT). Through extensive simulations of 5G NR-U/Wi-Fi coexistence scenarios, we show that QaSAL-CPM enforces 95th-percentile delay compliance and improves policy robustness. These results demonstrate that state-augmented constrained RL offers a scalable and adaptive solution for real-time coexistence optimization in dynamic wireless networks. 

   more » « less
3. RL-QN: A Reinforcement Learning Framework for Optimal Control of Queueing Systems

   https://doi.org/10.1145/3529375  

   Liu, Bai; Xie, Qiaomin; Modiano, Eytan (March 2022, ACM Transactions on Modeling and Performance Evaluation of Computing Systems)

   With the rapid advance of information technology, network systems have become increasingly complex and hence the underlying system dynamics are often unknown or difficult to characterize. Finding a good network control policy is of significant importance to achieve desirable network performance (e.g., high throughput or low delay). In this work, we consider using model-based reinforcement learning (RL) to learn the optimal control policy for queueing networks so that the average job delay (or equivalently the average queue backlog) is minimized. Traditional approaches in RL, however, cannot handle the unbounded state spaces of the network control problem. To overcome this difficulty, we propose a new algorithm, called RL for Queueing Networks (RL-QN), which applies model-based RL methods over a finite subset of the state space while applying a known stabilizing policy for the rest of the states. We establish that the average queue backlog under RL-QN with an appropriately constructed subset can be arbitrarily close to the optimal result. We evaluate RL-QN in dynamic server allocation, routing, and switching problems. Simulation results show that RL-QN minimizes the average queue backlog effectively. 

   more » « less
4. Constrained Reinforcement Learning for Fair and Environmentally Efficient Traffic Signal Controllers

   https://doi.org/10.1145/3676169  

   Haydari, Ammar; Aggarwal, Vaneet; Zhang, Michael; Chuah, Chen-Nee (March 2025, ACM Journal on Autonomous Transportation Systems)

   Traffic signal controller (TSC) has a crucial role in managing traffic flow in urban areas. Recently, reinforcement learning (RL) models have received a great attention for TSC with promising results. However, these RL-TSC models still need to be improved for real-world deployment due to limited exploration of different performance metrics such as fair traffic scheduling or air quality impact. In this work, we introduce a constrained multi-objective RL model that minimizes multiple constrained objectives while achieving a higher expected reward. Furthermore, our proposed RL strategy integrates the peak and average constraint models to the RL problem formulation with maximum entropy off-policy models. We applied this strategy to a single TSC and a network of TSCs. As part of this constrained RL-TSC formulation, we discuss fairness and air quality parameters as constraints for the closed-loop control system optimization model at TSCs calledFAirLight. Our experimental analysis shows that the proposedFAirLightachieves a good traffic flow performance in terms of average waiting time while being fair and environmentally friendly. Our method outperforms the baseline models and allows a more comprehensive view of RL-TSC regarding its applicability to the real world. 

   more » « less
5. Reinforcement Learning for Flooding Mitigation in Complex Stormwater Systems during Large Storms

   https://doi.org/10.1109/EUROCON52738.2021.9535587  

   Wang, Cheng; Bowes, Benjamin D.; Beling, Peter A.; Goodall, Jonathan L. (July 2021, IEEE EUROCON 2021 - 19th International Conference on Smart Technologies)

   Compared with capital improvement projects, real-time control of stormwater systems may be a more effective and efficient approach to address the increasing risk of flooding in urban areas. One way to automate the design process of control policies is through reinforcement learning (RL). Recently, RL methods have been applied to small stormwater systems and have demonstrated better performance over passive systems and simple rule-based strategies. However, it remains unclear how effective RL methods are for larger and more complex systems. Current RL-based control policies also suffer from poor convergence and stability, which may be due to large updates made by the underlying RL algorithm. In this study, we use the Proximal Policy Optimization (PPO) algorithm and develop control policies for a medium-sized stormwater system that can significantly mitigate flooding during large storm events. Our approach demonstrates good convergence behavior and stability, and achieves robust out-of-sample performance. 

   more » « less