An official website of the United States government
The integration of renewable energy resources has made power system management increasingly complex. DRL is a potential solution to optimize power system operations, but it requires significant time and resources during training. The control policies developed using DRL are specific to a single grid and require retraining from scratch for other grids. Training the DRL model from scratch is computationally expensive. This paper proposes a novel TL with a DRL framework to optimize VV C across different grids. This framework significantly reduces training time and improves VVC control performance by fine-tuning pre-trained DRL models for various grids. We developed a policy reuse classifier that transfers the knowledge from the IEEE-123 Bus system to the IEEE-13 Bus system. We performed an impact analysis to determine the effectiveness of TL. Our results show that TL improves the VVC control policy by 69.51 %, achieves faster convergence, and reduces the training time by 98.14%.
more »
« less
Implementing Deep Reinforcement Learning-Based Grid Voltage Control in Real-World Power Systems: Challenges and Insights
Deep reinforcement learning (DRL) holds significant promise for managing voltage control challenges in simulated power grid environments. However, its real-world application in power system operations remains underexplored. This study rigorously evaluates DRL’s performance and limitations within actual operational contexts by utilizing detailed experiments across the IEEE 14-bus system, Illinois 200-bus system, and the ISO New England node-breaker model. Our analysis critically assesses DRL’s effectiveness for grid control from a system operator's perspective, identifying specific performance bottlenecks. The findings provide actionable insights that highlight the necessity of advancing AI technologies to effectively address the growing complexities of modern power systems. This research underscores the vital role of DRL in enhancing grid management and reliability.
more »
« less
- PAR ID:
- 10614684
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3503-9042-1
- Page Range / eLocation ID:
- 1 to 5
- Subject(s) / Keyword(s):
- Deep reinforcement learning, autonomous voltage control, model fidelity, topology change.
- Format(s):
- Medium: X
- Location:
- Dubrovnik, Croatia
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
The high penetration of Renewable Energy Sources (RES) into the grid introduces complexity to the operation and optimization of energy. One potential solution to the challenge is to use deep reinforcement learning (DRL) based techniques to regulate voltage and reactive power under dynamic conditions. However, there is a need to optimize the DRL for better performance and robustness. This paper proposes a Bayesian optimization (BO) technique within the DRL framework to improve the performance and robustness of volt-var control (VVC) in power distribution systems. We combine the actor-critic DRL algorithm with the BO framework to yield fast optimal volt-var control policies. We use BO techniques to estimate DRL-based VVC decisions and accelerate model-training convergence. In the case study, we demonstrated that the BO in DRL on IEEE-13 has improved decision-making by 21.11% and 81.81% for 123 bus test systems. Our research shows that Bayesian-enabled DRL adapts to different grid configurations and maintains voltage profiles within desired limits, thereby improving DRL control policies.more » « less
-
The increasing penetration of Distributed Energy Resources (DERs) in modern distribution grids has created sig-nificant challenges in maintaining voltage profiles and minimizing reactive power losses. While Deep Reinforcement Learning (DRL) approaches have been explored for Volt/VAR Control (VVC), existing methods often rely on single-agent frameworks or lack adaptive mechanisms to balance competing grid objectives dy-namically. Multi-Agent (MA) DRL frameworks have emerged but typically require explicit inter-agent communication, limiting scalability and practicality. To address these limitations, we propose a novel MA-DRL framework that incorporates adaptive reward computation to implicitly coordinate special-ized agents, including a Voltage Control Agent for maintaining voltage profiles and a Reactive Power Agent for optimizing reactive power dispatch. Unlike prior approaches, our method dynamically adjusts local and global rewards based on real-time grid conditions, enabling scalable and efficient grid control without explicit inter-agent communication. We evaluated the framework on the IEEE 123-bus test system under high DER penetration. The results demonstrated that the proposed MA-DRL approach successfully eliminated voltage violations, reduced power losses through optimized reactive power dispatch, and decreased transformer tap changes and capacitor operations, thereby extending equipment lifespan. These results demonstrate the scalability and practicality of adaptive reward-based MA-DRL for addressing modern grid management challenges.more » « less
-
Integrating distributed energy resources (DERs) into a power system requires more advanced control mechanisms. One of the control strategies used for Volt-VAR control (VVC) is to manage voltage and reactive power. With the increase in the complexity of the power system, there is a need to develop an autonomous and robust control mechanism using deep reinforcement learning (DRL) to enhance grid performance and adjust voltage and reactive power settings. These adjustments minimize losses and enhance voltage stability in the grid. In this paper, we proposed a novel approach to develop a DRL-based VVC framework and mitigation techniques to protect against stealthy white-box attacks targeting the trained control policies of the DRL model. The mitigation technique on the trained DRL is proposed to control the voltage violations on the smart grid to enhance the stability of the grid and minimize voltage irregularities. Our proposed mitigation technique provided better control policies for DRL-based VVC, successfully mitigating 100 percent of voltage violations in the smart grid environment. The results show that the mitigation technique enhances the security and robustness of trained DRL VVC agents.more » « less
-
Abstract Deep Reinforcement Learning (DRL) has shown promise for voltage control in power systems due to its speed and model‐free nature. However, learning optimal control policies through trial and error on a real grid is infeasible due to the mission‐critical nature of power systems. Instead, DRL agents are typically trained on a simulator, which may not accurately represent the real grid. This discrepancy can lead to suboptimal control policies and raises concerns for power system operators. In this paper, we revisit the problem of RL‐based voltage control and investigate how model inaccuracies affect the performance of the DRL agent. Extensive numerical experiments are conducted to quantify the impact of model inaccuracies on learning outcomes. Specifically, techniques that enable the DRL agent are focused on learning robust policies that can still perform well in the presence of model errors. Furthermore, the impact of the agent's decisions on the overall system loss are analyzed to provide additional insight into the control problem. This work aims to address the concerns of power system operators and make DRL‐based voltage control more practical and reliable.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript
- Conference Paper:
- https://doi.org/10.1109/ISGTEUROPE62998.2024.10863452
