skip to main content


The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 5:00 PM ET until 11:00 PM ET on Friday, June 21 due to maintenance. We apologize for the inconvenience.

This content will become publicly available on November 7, 2024

Title: Optimal Control of District Cooling Energy Plant With Reinforcement Learning and MPC
We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the complexity of the DCEP model. Reinforcement learning (RL) is an attractive alternative since its real-time control computation is much simpler. But designing an RL controller is challenging due to myriad design choices and computationally intensive training. In this paper, we propose an RL controller and an MPC controller for minimizing the electricity cost of a DCEP and compare them via simulations. The two controllers are designed to be comparable in terms of objective and information requirements. The RL controller uses a novel Q-learning algorithm that is based on least-squares policy iteration. We describe the design choices for the RL controller, including the choice of state space and basis functions, that are found to be effective. The proposed MPC controller does not need a mixed integer solver for implementation, but only a nonlinear program (NLP) solver. A rule-based baseline controller is also proposed to aid in comparison. Simulation results show that the proposed RL and MPC controllers achieve similar savings over the baseline controller, about 17%.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ; ;
Publisher / Repository:
Date Published:
Journal Name:
ASME Journal of Engineering for Sustainable Buildings and Cities
Page Range / eLocation ID:
1 to 16
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    With increase in the frequency of natural disasters such as hurricanes that disrupt the supply from the grid, there is a greater need for resiliency in electric supply. Rooftop solar photovoltaic (PV) panels along with batteries can provide resiliency to a house in a blackout due to a natural disaster. Our previous work showed that intelligence can reduce the size of a PV+battery system for the same level of post-blackout service compared to a conventional system that does not employ intelligent control. The intelligent controller proposed is based on model predictive control (MPC), which has two main challenges. One, it requires simple yet accurate models as it involves real-time optimization. Two, the discrete actuation for residential loads (on/off) makes the underlying optimization problem a mixed-integer program (MIP) which is challenging to solve. An attractive alternative to MPC is reinforcement learning (RL) as the real-time control computation is both model-free and simple. These points of interest accompany certain trade-offs; RL requires computationally expensive offline learning, and its performance is sensitive to various design choices. In this work, we propose an RL-based controller. We compare its performance with the MPC controller proposed in our prior work and a non-intelligent baseline controller. The RL controller is found to provide a resiliency performance — by commanding critical loads and batteries—similar to MPC with a significant reduction in computational effort. 
    more » « less
  2. null (Ed.)
    Model predictive control (MPC) has been widely investigated for climate control of commercial buildings for both energy efficiency and demand flexibility. However, most MPC formulations ignore humidity and latent heat. The inclusion of moisture makes the problem considerably more challenging, primarily since a cooling and dehumidifying coil model which accounts for both sensible and latent heat transfers is needed. In our recent work, we proposed an MPC controller in which humidity and latent heat were incorporated in a principled manner, by using a reduced-order model of the cooling coil. Because of the highly nonlinear nature of the process in a cooling coil, the model needs to be modified based on certain weather/climatic conditions to have sufficient prediction accuracy. Doing so, however, leads to a mixed-integer nonlinear program (MINLP) that is challenging to solve. In this work, we propose an MPC formulation that retains the NLP (nonlinear programming problem) structure in all climate zones/weather conditions. This feature makes the control system capable of autonomous operation. Simulations in multiple climate zones and weather conditions verify the energy savings performance, and autonomy of the proposed controller. We also compare the performance of the proposed MPC controller with an MPC formulation that does not explicitly consider humidity. Under certain conditions, it is found that the MPC controller that excludes humidity leads to poor humidity control, or higher energy usage as it is unaware of the latent load on the cooling coil. 
    more » « less
  3. null (Ed.)
    In modern high-performance aircraft, the Fuel Thermal Management System (FTMS) plays a critical role in the overall thermal energy management of the aircraft. Actuator and state constraints in the FTMS limit the thermal endurance and capabilities of the aircraft. Thus, an effective control strategy must plan and execute optimized transient fuel mass and temperature trajectories subject to these constraints over the entire course of operation. For the control of linear systems, hierarchical Model Predictive Control (MPC) has shown to be an effective approach to coordinating both short- and long-term system operation with reduced computational complexity. However, for controlling nonlinear systems, common approaches to system linearization may no longer be effective due to the long prediction horizons of upper-level controllers. This paper explores the limitations of using linear models for hierarchical MPC of the nonlinear FTMS found in aircraft. Numerical simulation results show that linearized models work well for lower-level controllers with short prediction horizons but lead to significant reductions in aircraft thermal endurance when used for upper-level controllers with long prediction horizons. Therefore, a mixed-linearity hierarchical MPC formulation is presented with a nonlinear upper-level controller and a linear lower-level controller to achieve both high performance and high computational efficiency. 
    more » « less
  4. Abstract

    The ability to withstand and recover from disruptions is essential for seaport energy systems, and in light of the growing push for decarbonization, incorporating clean energy sources has become increasingly imperative to ensure resilience. This paper proposes a resilience enhancement planning strategy for a seaport multi‐energy system that integrates various energy modalities and sources, including heating, cooling, hydrogen, solar, and wind power. The planning strategy aims to ensure the reliable operation of the system during contingency events, such as power outages, equipment failures, or extreme weather incidents. The proposed optimization model is designed as a mixed‐integer nonlinear programming formulation, in which McCormick inequalities and other linearization techniques are utilized to tackle the model nonlinearities. The model allocates fuel cell electric trucks (FCETs), renewable energy sources, hydrogen refueling stations, and remote control switches such that the system resilience is enhanced while incorporating natural‐gas‐powered combined cooling, heating, and power system to minimize the operation and unserved demand costs. The model considers various factors such as the availability of renewable energy sources, the demand for heating, cooling, electricity, and hydrogen, the operation of remote control switches to help system reconfiguration, the travel behaviour of FCETs, and the power output of FCETs via vehicle‐to‐grid interface. The numerical results demonstrate that the proposed strategy can significantly improve the resilience of the seaport multi‐energy system and reduce the risk of service disruptions during contingency scenarios.

    more » « less
  5. Deep reinforcement learning approaches are becoming appealing for the design of nonlinear controllers for voltage control problems, but the lack of stability guarantees hinders their real-world deployment. This letter constructs a decentralized RL-based controller for inverter-based real-time voltage control in distribution systems. It features two components: a transient control policy and a steady-state performance optimizer. The transient policy is parameterized as a neural network, and the steady-state optimizer represents the gradient of the long-term operating cost function. The two parts are synthesized through a safe gradient flow framework, which prevents the violation of reactive power capacity constraints. We prove that if the output of the transient controller is bounded and monotonically decreasing with respect to its input, then the closed-loop system is asymptotically stable and converges to the optimal steady-state solution. We demonstrate the effectiveness of our method by conducting experiments with IEEE 13-bus and 123-bus distribution system test feeders. 
    more » « less