skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Explore Research Products in the PAR
It may take a few hours for recently added research products to appear in PAR search results.


Title: Optimal Control of District Cooling Energy Plant With Reinforcement Learning and MPC
We consider the problem of optimal control of district cooling energy plants (DCEPs) consisting of multiple chillers, a cooling tower, and a thermal energy storage (TES), in the presence of time-varying electricity price. A straightforward application of model predictive control (MPC) requires solving a challenging mixed-integer nonlinear program (MINLP) because of the on/off of chillers and the complexity of the DCEP model. Reinforcement learning (RL) is an attractive alternative since its real-time control computation is much simpler. But designing an RL controller is challenging due to myriad design choices and computationally intensive training. In this paper, we propose an RL controller and an MPC controller for minimizing the electricity cost of a DCEP and compare them via simulations. The two controllers are designed to be comparable in terms of objective and information requirements. The RL controller uses a novel Q-learning algorithm that is based on least-squares policy iteration. We describe the design choices for the RL controller, including the choice of state space and basis functions, that are found to be effective. The proposed MPC controller does not need a mixed integer solver for implementation, but only a nonlinear program (NLP) solver. A rule-based baseline controller is also proposed to aid in comparison. Simulation results show that the proposed RL and MPC controllers achieve similar savings over the baseline controller, about 17%.  more » « less
Award ID(s):
1934322
PAR ID:
10477145
Author(s) / Creator(s):
; ; ;
Publisher / Repository:
ASME
Date Published:
Journal Name:
ASME Journal of Engineering for Sustainable Buildings and Cities
ISSN:
2642-6641
Page Range / eLocation ID:
1 to 16
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. ; ; ; (, American Control Conference)
    null (Ed.)
    With increase in the frequency of natural disasters such as hurricanes that disrupt the supply from the grid, there is a greater need for resiliency in electric supply. Rooftop solar photovoltaic (PV) panels along with batteries can provide resiliency to a house in a blackout due to a natural disaster. Our previous work showed that intelligence can reduce the size of a PV+battery system for the same level of post-blackout service compared to a conventional system that does not employ intelligent control. The intelligent controller proposed is based on model predictive control (MPC), which has two main challenges. One, it requires simple yet accurate models as it involves real-time optimization. Two, the discrete actuation for residential loads (on/off) makes the underlying optimization problem a mixed-integer program (MIP) which is challenging to solve. An attractive alternative to MPC is reinforcement learning (RL) as the real-time control computation is both model-free and simple. These points of interest accompany certain trade-offs; RL requires computationally expensive offline learning, and its performance is sensitive to various design choices. In this work, we propose an RL-based controller. We compare its performance with the MPC controller proposed in our prior work and a non-intelligent baseline controller. The RL controller is found to provide a resiliency performance — by commanding critical loads and batteries—similar to MPC with a significant reduction in computational effort. 
    more » « less
  2. ; ; (, 2020-2021 International Conference on High Performance Buildings)
    null (Ed.)
    Model predictive control (MPC) has been widely investigated for climate control of commercial buildings for both energy efficiency and demand flexibility. However, most MPC formulations ignore humidity and latent heat. The inclusion of moisture makes the problem considerably more challenging, primarily since a cooling and dehumidifying coil model which accounts for both sensible and latent heat transfers is needed. In our recent work, we proposed an MPC controller in which humidity and latent heat were incorporated in a principled manner, by using a reduced-order model of the cooling coil. Because of the highly nonlinear nature of the process in a cooling coil, the model needs to be modified based on certain weather/climatic conditions to have sufficient prediction accuracy. Doing so, however, leads to a mixed-integer nonlinear program (MINLP) that is challenging to solve. In this work, we propose an MPC formulation that retains the NLP (nonlinear programming problem) structure in all climate zones/weather conditions. This feature makes the control system capable of autonomous operation. Simulations in multiple climate zones and weather conditions verify the energy savings performance, and autonomy of the proposed controller. We also compare the performance of the proposed MPC controller with an MPC formulation that does not explicitly consider humidity. Under certain conditions, it is found that the MPC controller that excludes humidity leads to poor humidity control, or higher energy usage as it is unaware of the latent load on the cooling coil. 
    more » « less
  3. ; (, Dynamic Systems and Control Conference)
    null (Ed.)
    In modern high-performance aircraft, the Fuel Thermal Management System (FTMS) plays a critical role in the overall thermal energy management of the aircraft. Actuator and state constraints in the FTMS limit the thermal endurance and capabilities of the aircraft. Thus, an effective control strategy must plan and execute optimized transient fuel mass and temperature trajectories subject to these constraints over the entire course of operation. For the control of linear systems, hierarchical Model Predictive Control (MPC) has shown to be an effective approach to coordinating both short- and long-term system operation with reduced computational complexity. However, for controlling nonlinear systems, common approaches to system linearization may no longer be effective due to the long prediction horizons of upper-level controllers. This paper explores the limitations of using linear models for hierarchical MPC of the nonlinear FTMS found in aircraft. Numerical simulation results show that linearized models work well for lower-level controllers with short prediction horizons but lead to significant reductions in aircraft thermal endurance when used for upper-level controllers with long prediction horizons. Therefore, a mixed-linearity hierarchical MPC formulation is presented with a nonlinear upper-level controller and a linear lower-level controller to achieve both high performance and high computational efficiency. 
    more » « less
  4. ; ; ; ; (, IEEE)
    Designing robust algorithms in the face of estimation uncertainty is a challenging task. Indeed, controllers seldom consider estimation uncertainty and only rely on the most likely estimated state. Consequently, sudden changes in the environment or the robot’s dynamics can lead to catastrophic behaviors. Leveraging recent results in risk-sensitive optimal control, this paper presents a risk-sensitive Extended Kalman Filter that can adapt its estimation to the control objective, hence allowing safe output-feedback Model Predictive Control (MPC). By taking a pessimistic estimate of the value function resulting from the MPC controller, the filter provides increased robustness to the controller in phases of uncertainty as compared to a standard Extended Kalman Filter (EKF). The filter has the same computational complexity as an EKF and can be used for real-time control. The paper evaluates the risk-sensitive behavior of the proposed filter when used in a nonlinear MPC loop on a planar drone and industrial manipulator in simulation, as well as on an external force estimation task on a real quadruped robot. These experiments demonstrate the ability of the approach to significantly improve performance in face of uncertainties. 
    more » « less
  5. ; ; (, IET Generation, Transmission & Distribution)
    Abstract The ability to withstand and recover from disruptions is essential for seaport energy systems, and in light of the growing push for decarbonization, incorporating clean energy sources has become increasingly imperative to ensure resilience. This paper proposes a resilience enhancement planning strategy for a seaport multi‐energy system that integrates various energy modalities and sources, including heating, cooling, hydrogen, solar, and wind power. The planning strategy aims to ensure the reliable operation of the system during contingency events, such as power outages, equipment failures, or extreme weather incidents. The proposed optimization model is designed as a mixed‐integer nonlinear programming formulation, in which McCormick inequalities and other linearization techniques are utilized to tackle the model nonlinearities. The model allocates fuel cell electric trucks (FCETs), renewable energy sources, hydrogen refueling stations, and remote control switches such that the system resilience is enhanced while incorporating natural‐gas‐powered combined cooling, heating, and power system to minimize the operation and unserved demand costs. The model considers various factors such as the availability of renewable energy sources, the demand for heating, cooling, electricity, and hydrogen, the operation of remote control switches to help system reconfiguration, the travel behaviour of FCETs, and the power output of FCETs via vehicle‐to‐grid interface. The numerical results demonstrate that the proposed strategy can significantly improve the resilience of the seaport multi‐energy system and reduce the risk of service disruptions during contingency scenarios. 
    more » « less
  • Citation Formats
  • MLA
  • APA
  • Chicago
  • BibTeX
  • Save / Share this Record
  • Send to Email