skip to main content


The NSF Public Access Repository (NSF-PAR) system and access will be unavailable from 5:00 PM ET until 11:00 PM ET on Friday, June 21 due to maintenance. We apologize for the inconvenience.

Title: Non-intrusive interpretation of human thermal comfort through analysis of facial infrared thermography
Understanding occupants’ thermal sensation and comfort is essential to defining the operational settings for Heating, Ventilation and Air Conditioning (HVAC) systems in buildings. Due to the continuous impact of human and environmental factors, occupants’ thermal sensation and comfort level can change over time. Thus, to dynamically control the environment, thermal comfort should be monitored in real time. This paper presents a novel non-intrusive infrared thermography framework to estimate an occupant’s thermal comfort level by measuring skin temperature collected from different facial regions using low- cost thermal cameras. Unlike existing methods that rely on placing sensors directly on humans for skin temperature measurement, the proposed framework is able to detect the presence of occupants, extract facial regions, measure skin temperature features, and interpret thermal comfort conditions with minimal interruption of the building occupants. The method is validated by collecting thermal comfort data from a total of twelve subjects under cooling, heating and steady-state experiments. The results demonstrate that ears, nose and cheeks are most indicative of thermal comfort and the proposed framework can be used to assess occupants’ thermal comfort with an average accuracy of 85%.  more » « less
Award ID(s):
Author(s) / Creator(s):
; ;
Date Published:
Journal Name:
Energy and buildings
Page Range / eLocation ID:
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Understanding occupants’ thermal comfort is essential for the effective operation of Heating, Ventilation, and Air Conditioning (HVAC) systems. Existing studies of the “human-in-the-loop” HVAC control generally suffer from: (1) excessive reliance on cumbersome human feedback; and (2) intrusiveness caused by conventional data collection methods. To address these limitations, this paper investigates the low-cost thermal camera as a non-intrusive approach to assess thermal comfort in real time using facial skin temperature. The framework developed can automatically detect occupants, extract facial regions, measure skin temperature, and interpret thermal comfort with minimal interruption or participation of occupants. The framework is validated using the facial skin temperature collected from twelve occupants. Personal comfort models trained from different machine learning algorithms are compared and results show that Random Forest model can achieve an accuracy of 85% and also suggest that the skin temperature of ears, nose, and cheeks are most indicative of thermal comfort. 
    more » « less
  2. It is well established that thermal comfort is an influential factor in human health and wellbeing. Uncomfortable thermal environments can reduce occupants’ comfort and productivity, and cause symptoms of sick building syndrome. To harness the built environment as a medium to support human health, well-being, and engagement, it is significantly important to understand occupants’ thermal comfort in real time. To this end, this study proposes a non-intrusive method to collect occupants’ facial skin temperature and interpret their thermal comfort conditions by fusing the thermal and RGB-D images collected from multiple low-cost thermographic and Kinect sensors. This study distinguishes from existing methods of thermal comfort assessment in three ways: 1) it is a truly non-intrusive data collection approach which has a minimal interruption or participation of building occupants; 2) the proposed approach can simultaneously identify and interpret multiple occupants’ thermal comfort; 3) it uses low-cost thermographic and RGB-D cameras which can be rapidly deployed and reconfigured to adapt to various settings. This approach was experimentally evaluated in a transient heating environment (room temperature increased from 23 to 27 °C) to verify its applicability in real operational built environments. In total, all 6 subjects observed moderate to strong positive correlations between the ambient room temperature and subjects’ facial skin temperature collected using the proposed approach. Additionally, all 6 subjects have voted different thermal sensations at the beginning (the first 5 minutes) and at the end (the last 5 minutes) of the heating experiment, which can be reflected by the significant differences in the mean skin temperature of these two periods (p < .001). Results of this pilot study demonstrate the feasibility of applying the proposed non-intrusive approach to real multi-occupancy environments to dynamically interpret occupants’ thermal comfort and optimize the operation of building heating, ventilation and air conditioning (HVAC) systems. 
    more » « less
  3. null (Ed.)
    Mental workload represents the mental resources an individual devotes to a task. In a building environment, understanding how ambient thermal conditions affect occupants' mental workload offers an opportunity to achieve optimal thermal settings for the heating, ventilation, and air conditioning (HVAC) systems. However, directly measuring mental workload on a large and continuous scale requires occupants to perform subjective tests or wear electroencephalogram (EEG) or similar devices, which is impractical. This paper assesses the feasibility of using infrared facial thermography captured by a low-cost thermal camera to disclose mental workload. An experiment was conducted to measure the facial skin temperature while subjects performed cognitive tasks in three different thermal environments, representing occupants' thermal sensation of slightly cool, neutral, and slightly warm. Mental workload was measured using an EEG headset to eliminate subjective bias. The correlations between facial temperature and mental workload vary with different individuals and thermal conditions. Relatively strong correlations are found in the neutral environment and in the regions of ears, mouth, and neck. The results also suggest that future work should collect data under extended experiment duration. This is because it was observed that the response of facial skin temperature to mental workload varies with task type; thus, increasing the repetitiveness for each type of task or using more challenging tasks in the experiment could potentially lead to more insights on this relationship. 
    more » « less
  4. About 40% of the energy produced globally is consumed within buildings, primarily for providing occupants with comfortable work and living spaces. However, despite the significant impacts of such energy consumption on the environment, the lack of thermal comfort among occupants is a common problem that can lead to health complications and reduced productivity. To address this problem, it is particularly important to understand occupants’ thermal comfort in real-time to dynamically control the environment. This study investigates an infrared thermal camera network to extract skin temperature features and predict occupants’ thermal preferences at flexible distances and angles. This study distinguishes from existing methods in two ways: (1) the proposed method is a non-intrusive data collection approach which does not require human participation or personal devices; (2) it uses low-cost thermal cameras and RGB-D sensors which can be rapidly reconfigured to adapt to various settings and has little or no hardware infrastructure dependency. The proposed camera network is verified using the facial skin temperature collected from 16 subjects in a multi-occupancy experiment. The results show that all 16 subjects observed a statistically higher skin temperature as the room temperature increases. The variations in skin temperature also correspond to the distinct comfort states reported by the subjects. The post-experiment evaluation suggests that the networked thermal cameras have a minimal interruption of building occupants. The proposed approach demonstrates the potential to transition the human physiological data collection from an intrusive and wearable device-based approach to a truly non-intrusive and scalable approach. 
    more » « less
  5. Rooftop photovoltaics (PV) and electrical vehicles (EV) have become more economically viable to residential customers. Most existing home energy management systems (HEMS) only focus on the residential occupants’ thermal comfort in terms of indoor temperature and humidity while neglecting their other behaviors or concerns. This paper aims to integrate residential PV and EVs into the HEMS in an occupant-centric manner while taking into account the occupants’ thermal comfort, clothing behaviors, and concerns on the state-of-charge (SOC) of EVs. A stochastic adaptive dynamic programming (ADP) model was proposed to optimally determine the setpoints of heating, ventilation, air conditioning (HVAC), occupant’s clothing decisions, and the EV’s charge/discharge schedule while considering uncertainties in the outside temperature, PV generation, and EV’s arrival SOC. The nonlinear and nonconvex thermal comfort model, EV SOC concern model, and clothing behavior model were holistically embedded in the ADP-HEMS model. A model predictive control framework was further proposed to simulate a residential house under the time of use tariff, such that it continually updates with optimal appliance schedules decisions passed to the house model. Cosimulations were carried out to compare the proposed HEMS with a baseline model that represents the current operational practice. The result shows that the proposed HEMS can reduce the energy cost by 68.5% while retaining the most comfortable thermal level and negligible EV SOC concerns considering the occupant’s behaviors. 
    more » « less