skip to main content


Title: Can Infrared Facial Thermography Disclose Mental Workload in Indoor Thermal Environments?
Mental workload represents the mental resources an individual devotes to a task. In a building environment, understanding how ambient thermal conditions affect occupants' mental workload offers an opportunity to achieve optimal thermal settings for the heating, ventilation, and air conditioning (HVAC) systems. However, directly measuring mental workload on a large and continuous scale requires occupants to perform subjective tests or wear electroencephalogram (EEG) or similar devices, which is impractical. This paper assesses the feasibility of using infrared facial thermography captured by a low-cost thermal camera to disclose mental workload. An experiment was conducted to measure the facial skin temperature while subjects performed cognitive tasks in three different thermal environments, representing occupants' thermal sensation of slightly cool, neutral, and slightly warm. Mental workload was measured using an EEG headset to eliminate subjective bias. The correlations between facial temperature and mental workload vary with different individuals and thermal conditions. Relatively strong correlations are found in the neutral environment and in the regions of ears, mouth, and neck. The results also suggest that future work should collect data under extended experiment duration. This is because it was observed that the response of facial skin temperature to mental workload varies with task type; thus, increasing the repetitiveness for each type of task or using more challenging tasks in the experiment could potentially lead to more insights on this relationship.  more » « less
Award ID(s):
1804321
NSF-PAR ID:
10199586
Author(s) / Creator(s):
; ; ;
Date Published:
Journal Name:
UrbSys'19: Proceedings of the 1st ACM International Workshop on Urban Building Energy Sensing, Controls, Big Data Analysis, and Visualization
Page Range / eLocation ID:
87-96
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. Workers' performance in indoor offices can be greatly affected by the thermal condition of the environment. However, this effect can be difficult to quantify, especially when the thermal stress is a moderate increase or decrease in temperature and the work productivity cannot be directly measured. Subjects' high motivation to perform well under experimental conditions also causes difficulties in comparing their performance in different thermal environments. In order to overcome these limitations, this paper proposes a method to investigate the effect of the indoor thermal conditions on occupants' performance by studying occupants' mental workload measured by the electroencephalography (EEG) when they perform standardized cognitive tasks. An experiment integrating EEG mental workload measurement and cognitive tasks was implemented on 15 subjects. EEG data were collected while subjects were performing four cognitive tasks on computers. Based on previous studies, we propose a mental workload index calculated from the frontal theta and parietal alpha frequency band power. Within-subject comparisons were performed to investigate whether subjects' mental workload is statistically different under three different thermal environments, representing thermal sensations of slightly cool, neutral, and slightly warm. The results show that the effect of thermal environment varies across different individuals. By comparing the mental workload index among different thermal environments, we found that the slightly warm environment resulted in a relatively higher mental workload than the other two environments to achieve the same performance. The study provides promising insights into how the thermal environment influences occupants’ performance by affecting their mental workload from the neurophysiological perspective. 
    more » « less
  2. The thermal environment has a great influence on individuals’ performance; however, factors such as one’s motivation to perform well under experimental conditions cause difficulties in assessing how room temperature affect subjects’ performance. One approach to overcome this problem is to understand the changes in individuals’ neurophysiological conditions. This paper reports on the results of an experiment where electroencephalogram (EEG) data were collected from 5 subjects while they performed four computerized cognitive tasks. Power spectral density of EEG signals in three different thermal environments, slightly cool, neutral, and slightly warm, was compared within subjects. In most cases, significant differences in PSD of the frontal theta (4–8 Hz) activity are observed, indicating individuals’ mental effort varies with room temperature. In the long run, the increased mental workload will reduce individuals’ performance and be detrimental to their productivity. The study indicates that the proposed method could be implemented on a larger scale for further studies. 
    more » « less
  3. It is well established that thermal comfort is an influential factor in human health and wellbeing. Uncomfortable thermal environments can reduce occupants’ comfort and productivity, and cause symptoms of sick building syndrome. To harness the built environment as a medium to support human health, well-being, and engagement, it is significantly important to understand occupants’ thermal comfort in real time. To this end, this study proposes a non-intrusive method to collect occupants’ facial skin temperature and interpret their thermal comfort conditions by fusing the thermal and RGB-D images collected from multiple low-cost thermographic and Kinect sensors. This study distinguishes from existing methods of thermal comfort assessment in three ways: 1) it is a truly non-intrusive data collection approach which has a minimal interruption or participation of building occupants; 2) the proposed approach can simultaneously identify and interpret multiple occupants’ thermal comfort; 3) it uses low-cost thermographic and RGB-D cameras which can be rapidly deployed and reconfigured to adapt to various settings. This approach was experimentally evaluated in a transient heating environment (room temperature increased from 23 to 27 °C) to verify its applicability in real operational built environments. In total, all 6 subjects observed moderate to strong positive correlations between the ambient room temperature and subjects’ facial skin temperature collected using the proposed approach. Additionally, all 6 subjects have voted different thermal sensations at the beginning (the first 5 minutes) and at the end (the last 5 minutes) of the heating experiment, which can be reflected by the significant differences in the mean skin temperature of these two periods (p < .001). Results of this pilot study demonstrate the feasibility of applying the proposed non-intrusive approach to real multi-occupancy environments to dynamically interpret occupants’ thermal comfort and optimize the operation of building heating, ventilation and air conditioning (HVAC) systems. 
    more » « less
  4. Understanding occupants’ thermal sensation and comfort is essential to defining the operational settings for Heating, Ventilation and Air Conditioning (HVAC) systems in buildings. Due to the continuous impact of human and environmental factors, occupants’ thermal sensation and comfort level can change over time. Thus, to dynamically control the environment, thermal comfort should be monitored in real time. This paper presents a novel non-intrusive infrared thermography framework to estimate an occupant’s thermal comfort level by measuring skin temperature collected from different facial regions using low- cost thermal cameras. Unlike existing methods that rely on placing sensors directly on humans for skin temperature measurement, the proposed framework is able to detect the presence of occupants, extract facial regions, measure skin temperature features, and interpret thermal comfort conditions with minimal interruption of the building occupants. The method is validated by collecting thermal comfort data from a total of twelve subjects under cooling, heating and steady-state experiments. The results demonstrate that ears, nose and cheeks are most indicative of thermal comfort and the proposed framework can be used to assess occupants’ thermal comfort with an average accuracy of 85%. 
    more » « less
  5. About 40% of the energy produced globally is consumed within buildings, primarily for providing occupants with comfortable work and living spaces. However, despite the significant impacts of such energy consumption on the environment, the lack of thermal comfort among occupants is a common problem that can lead to health complications and reduced productivity. To address this problem, it is particularly important to understand occupants’ thermal comfort in real-time to dynamically control the environment. This study investigates an infrared thermal camera network to extract skin temperature features and predict occupants’ thermal preferences at flexible distances and angles. This study distinguishes from existing methods in two ways: (1) the proposed method is a non-intrusive data collection approach which does not require human participation or personal devices; (2) it uses low-cost thermal cameras and RGB-D sensors which can be rapidly reconfigured to adapt to various settings and has little or no hardware infrastructure dependency. The proposed camera network is verified using the facial skin temperature collected from 16 subjects in a multi-occupancy experiment. The results show that all 16 subjects observed a statistically higher skin temperature as the room temperature increases. The variations in skin temperature also correspond to the distinct comfort states reported by the subjects. The post-experiment evaluation suggests that the networked thermal cameras have a minimal interruption of building occupants. The proposed approach demonstrates the potential to transition the human physiological data collection from an intrusive and wearable device-based approach to a truly non-intrusive and scalable approach. 
    more » « less