skip to main content


This content will become publicly available on July 1, 2024

Title: Fast Human-in-the-Loop Control for HVAC Systems via Meta-Learning and Model-Based Offline Reinforcement Learning
Reinforcement learning (RL) methods can be used to develop a controller for the heating, ventilation, and air conditioning (HVAC) systems that both saves energy and ensures high occupants’ thermal comfort levels. However, the existing works typically require on-policy data to train an RL agent, and the occupants’ personalized thermal preferences are not considered, which is limited in the real-world scenarios. This paper designs a high-performance model-based offline RL algorithm for personalized HVAC systems. The proposed algorithm can quickly adapt to different occupants’ thermal preferences with a few thermal feedbacks, guaranteeing the high occupants’ personalized thermal comfort levels efficiently. First, we use a meta-supervised learning algorithm to train an occupant's thermal preference model. Then, we train an ensemble neural network to predict the thermal states of the considered zone. In addition, the obtained ensemble networks can indicate the regions in the state and action spaces covered by the offline dataset. With the personalized thermal preference model updated via meta-testing, model-based RL is used to derive the optimal HVAC controller. Since the proposed algorithm only requires offline datasets and a few online thermal feedbacks for training, it contributes to a more practical deployment of the RL algorithm to HVAC systems. We use the ASHRAE database II to verify the effectiveness and advantage of the meta-learning algorithm for modeling different occupants’ thermal preferences. Numerical simulations on the EnergyPlus environment demonstrate that the proposed algorithm can guarantee personalized thermal preferences with a slight increase of power consumption of 1.91% compared with the model-based RL algorithm with on-policy data aggregation.  more » « less
Award ID(s):
1837021
NSF-PAR ID:
10467763
Author(s) / Creator(s):
; ;
Publisher / Repository:
Institute of Electrical and Electronics Engineers (IEEE)
Date Published:
Journal Name:
IEEE Transactions on Sustainable Computing
Volume:
8
Issue:
3
ISSN:
2377-3790
Page Range / eLocation ID:
504 to 521
Subject(s) / Keyword(s):
["HVAC","Data models","Computational modeling","Prediction algorithms","Atmospheric modeling","Optimization","Buildings"]
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. The model of personalized thermal comfort can be learned via various machine learning algorithms and used to improve the individuals’ thermal comfort levels with potentially less energy consumption of HVAC systems. However, the learning of such a model typically requires a substantial number of thermal votes from the considered occupant, and the environmental conditions needed for collecting some votes may be undesired by the occupant in order to obtain a model with good generalization ability. In this paper, we propose to use a meta-learning algorithm to reduce the required number of personalized thermal votes so that a personalized thermal comfort model can be obtained with only a small number of feedback. With the learned meta-model, we derive a method based on the backpropagation of neural networks to quickly identify the best environmental and personal conditions for a specific occupant. The proposed identification algorithm has an additional advantage that the thermal comfort, indicated by the mean thermal sensation value, improves incrementally during the data collection process. We use the ASHRAE global thermal comfort database II to verify that the meta-learning algorithm can achieve an improved prediction accuracy after using 5 thermal sensation votes from an occupant to make adaptations. In addition, we show the effectiveness of the fast identification algorithm for the best personalized thermal environmental conditions with a thermal sensation generation model built from the PMV model. 
    more » « less
  2. Thermal comfort and energy efficiency are always the two most significant objectives in HVAC operations. However, for conventional HVAC systems, the pursuit of high energy efficiency may be at the expense of satisfactory thermal comfort. Therefore, even if centralized HVAC systems nowadays have higher energy efficiency than before in office buildings, most of them cannot adapt the dynamic occupant behaviors or individual thermal comfort. In order to realize high energy efficiency while still maintain satisfactory thermal environment for occupants indoors, the integrated hybrid HVAC system has been developed for years such as task-ambient conditioning system. Moreover, the occupant-based HVAC control system such as human- in-the-loop has also been investigated so that the system can be adaptive based on occupant behaviors. However, most of research related to personalized air-conditioning system only focuses on field-study with limited scale (i.e. only one office room), this paper has proposed a co- simulation model in energyplus to simulate the hybrid cooling system with synthetic thermal comfort distributions based on global comfort database I&II. An optimization framework on cooling set-point is proposed with the objective of energy performance and the constraints of thermal comfort distribution developed by unsupervised Gaussian mixture model (GMM) clustering and kernel density estimation (KDE). The co-simulation results have illustrated that with the proposed optimization algorithm and the hybrid cooling system, HVAC demand power has decreased 5.3% on average with at least 90% of occupants feeling satisfied. 
    more » « less
  3. Thermal comfort is a significant factor in the indoor building environment because it influences both human productivity and health. A currently popular method for predicting thermal comfort levels, the Predicted Mean Vote (PMV) and Predicted Percent Dissatisfied (PPD) model, unfortunately, has certain limitations. Consequently, the development of a better method for making accurate predictions (especially for individuals) is needed. Our goal was to develop a tool to predict individual thermal comfort preferences and automatically control the heating, ventilation, and air conditioning (HVAC) systems. This study adopted a series of human-subject experiments to collect essential data. All collected data was analyzed by adopting different machine learning algorithms. The machine learning algorithms predicted individual thermal comfort levels and thermal sensations, based on facial skin temperatures of participants in the experiments. These predictions were input data for the HVAC system control model, and results supported the potential for using facial skin temperatures to predict thermal comfort and thermal sensation levels. Moreover, this tool provided automatic control of the HVAC systems that can help improve the indoor environment of a building. 
    more » « less
  4. null (Ed.)
    In recent years, physiological features have gained more attention in developing models of personal thermal comfort for improved and accurate adaptive operation of Human-In-The-Loop (HITL) Heating, Ventilation, and Air-Conditioning (HVAC) systems. Pursuing the identification of effective physiological sensing systems for enhancing flexibility of human-centered and distributed control, using machine learning algorithms, we have investigated how heat flux sensing could improve personal thermal comfort inference under transient ambient conditions. We have explored the variations of heat exchange rates of facial and wrist skin. These areas are often exposed in indoor environments and contribute to the thermoregulation mechanism through skin heat exchange, which we have coupled with variations of skin and ambient temperatures for inference of personal thermal preferences. Adopting an experimental and data analysis methodology, we have evaluated the modeling of personal thermal preference of 18 human subjects for well-known classifiers using different scenarios of learning. The experimental measurements have revealed the differences in personal thermal preferences and how they are reflected in physiological variables. Further, we have shown that heat exchange rates have high potential in improving the performance of personal inference models even compared to the use of skin temperature. 
    more » « less
  5. Understanding occupants’ thermal comfort is essential for the effective operation of Heating, Ventilation, and Air Conditioning (HVAC) systems. Existing studies of the “human-in-the-loop” HVAC control generally suffer from: (1) excessive reliance on cumbersome human feedback; and (2) intrusiveness caused by conventional data collection methods. To address these limitations, this paper investigates the low-cost thermal camera as a non-intrusive approach to assess thermal comfort in real time using facial skin temperature. The framework developed can automatically detect occupants, extract facial regions, measure skin temperature, and interpret thermal comfort with minimal interruption or participation of occupants. The framework is validated using the facial skin temperature collected from twelve occupants. Personal comfort models trained from different machine learning algorithms are compared and results show that Random Forest model can achieve an accuracy of 85% and also suggest that the skin temperature of ears, nose, and cheeks are most indicative of thermal comfort. 
    more » « less