More Like this

1. Flood mitigation in coastal urban catchments using real-time stormwater infrastructure control and reinforcement learning

   https://doi.org/10.2166/hydro.2020.080  

   Bowes, Benjamin D. ; Tavakoli, Arash ; Wang, Cheng ; Heydarian, Arsalan ; Behl, Madhur ; Beling, Peter A. ; Goodall, Jonathan L. ( October 2020 , Journal of Hydroinformatics)

   null (Ed.)

   Abstract Flooding in coastal cities is increasing due to climate change and sea-level rise, stressing the traditional stormwater systems these communities rely on. Automated real-time control (RTC) of these systems can improve performance, and creating control policies for smart stormwater systems is an active area of study. This research explores reinforcement learning (RL) to create control policies to mitigate flood risk. RL is trained using a model of hypothetical urban catchments with a tidal boundary and two retention ponds with controllable valves. RL's performance is compared to the passive system, a model predictive control (MPC) strategy, and a rule-based control strategy (RBC). RL learns to proactively manage pond levels using current and forecast conditions and reduced flooding by 32% over the passive system. Compared to the MPC approach using a physics-based model and genetic algorithm, RL achieved nearly the same flood reduction, just 3% less than MPC, with a significant 88× speedup in runtime. Compared to RBC, RL was able to quickly learn similar control strategies and reduced flooding by an additional 19%. This research demonstrates that RL can effectively control a simple system and offers a computationally efficient method that could scale to RTC of more complex stormwater systems. 

   more » « less
2. Deep Reinforcement Learning with Uncertain Data for Real-Time Stormwater System Control and Flood Mitigation

   https://doi.org/10.3390/w12113222  

   Saliba, Sami M. ; Bowes, Benjamin D. ; Adams, Stephen ; Beling, Peter A. ; Goodall, Jonathan L. ( November 2020 , Water)

   null (Ed.)

   Flooding in many areas is becoming more prevalent due to factors such as urbanization and climate change, requiring modernization of stormwater infrastructure. Retrofitting standard passive systems with controllable valves/pumps is promising, but requires real-time control (RTC). One method of automating RTC is reinforcement learning (RL), a general technique for sequential optimization and control in uncertain environments. The notion is that an RL algorithm can use inputs of real-time flood data and rainfall forecasts to learn a policy for controlling the stormwater infrastructure to minimize measures of flooding. In real-world conditions, rainfall forecasts and other state information are subject to noise and uncertainty. To account for these characteristics of the problem data, we implemented Deep Deterministic Policy Gradient (DDPG), an RL algorithm that is distinguished by its capability to handle noise in the input data. DDPG implementations were trained and tested against a passive flood control policy. Three primary cases were studied: (i) perfect data, (ii) imperfect rainfall forecasts, and (iii) imperfect water level and forecast data. Rainfall episodes (100) that caused flooding in the passive system were selected from 10 years of observations in Norfolk, Virginia, USA; 85 randomly selected episodes were used for training and the remaining 15 unseen episodes served as test cases. Compared to the passive system, all RL implementations reduced flooding volume by 70.5% on average, and performed within a range of 5%. This suggests that DDPG is robust to noisy input data, which is essential knowledge to advance the real-world applicability of RL for stormwater RTC. 

   more » « less
3. Reinforcement Learning for Flooding Mitigation in Complex Stormwater Systems during Large Storms

   https://doi.org/10.1109/EUROCON52738.2021.9535587  

   Wang, Cheng ; Bowes, Benjamin D. ; Beling, Peter A. ; Goodall, Jonathan L. ( July 2021 , IEEE EUROCON 2021 - 19th International Conference on Smart Technologies)

   Compared with capital improvement projects, real-time control of stormwater systems may be a more effective and efficient approach to address the increasing risk of flooding in urban areas. One way to automate the design process of control policies is through reinforcement learning (RL). Recently, RL methods have been applied to small stormwater systems and have demonstrated better performance over passive systems and simple rule-based strategies. However, it remains unclear how effective RL methods are for larger and more complex systems. Current RL-based control policies also suffer from poor convergence and stability, which may be due to large updates made by the underlying RL algorithm. In this study, we use the Proximal Policy Optimization (PPO) algorithm and develop control policies for a medium-sized stormwater system that can significantly mitigate flooding during large storm events. Our approach demonstrates good convergence behavior and stability, and achieves robust out-of-sample performance. 

   more » « less
4. Data-Driven Model Predictive Control for Real-Time Stormwater Management

   Ning, Jingyun ; Bowes, Benjamin ; Goodall, Jonathan ; Behl, Madhur ( June 2022 , 2022 American Control Conference)

   Low-lying coastal cities across the world are increasingly seeing flooding due to climate change and accompanying sea-level rise. Many such cities rely on old and passive stormwater infrastructure which cannot cope up with the increasing flood risk. One potential solution for addressing coastal flooding is implementing active control strategies in stormwater systems. Active stormwater control relies on rule-based strategies, which is not able to manage the increasing flood risk. Model predictive control (MPC) for stormwater flood management is getting attention over the past decade. However, building physics-based models for MPC in stormwater management is cost and time prohibitive. In this paper, we develop a data-driven approach, which utilizes unstructured state-space models for system identification and predictive control implementation. We demonstrate our results using two real stormwater network configurations, one from the Norfolk, VA region and another model of Ann Arbor region, MI, respectively. Our results indicate that MPC outperforms rule-based strategies by up to 60% of the Norfolk model and up to 90% of the Ann Arbor model in flood management. 

   more » « less
5. Leveraging Open Source Software and Parallel Computing for Model Predictive Control Simulation of Urban Drainage Systems Using EPA-SWMM5 and Python

   https://doi.org/10.1007/978-3-319-99867-1_170  

   Sadler J.M., Goodall J.L. ( January 2019 , UDM 2018: New Trends in Urban Drainage Modelling)

   The active control of stormwater systems is a potential solution to increased street flooding in low-lying, low-relief coastal cities due to climate change and accompanying sea level rise. Model predictive control (MPC) has been shown to be a successful control strategy generally and as well as for managing urban drainage specifically. This research describes and demonstrates the implementation of MPC for urban drainage systems using open source software (Python and The United States Environmental Protection Agency (EPA) Storm Water Management Model (SWMM5). The system was demonstrated using a simplified use case in which an actively-controlled outlet of a detention pond is simulated. The control of the pond’s outlet influences the flood risk of a downstream node. For each step in the SWMM5 model, a series of policies for controlling the outlet are evaluated. The best policy is then selected using an evolutionary algorithm. The policies are evaluated against an objective function that penalizes primarily flooding and secondarily deviation of the detention pond level from a target level. Freely available Python libraries provide the key functionality for the MPC workflow: step-by-step running of the SWMM5 simulation, evolutionary algorithm implementation, and leveraging parallel computing. For perspective, the MPC results were compared to results from a rule-based approach and a scenario with no active control. The MPC approach produced a control policy that largely eliminated flooding (unlike the scenario with no active control) and maintained the detention pond’s water level closer to a target level (unlike the rule-based approach). 

   more » « less