skip to main content

Title: Using Machine Learning to Train a Wearable Device for Measuring Students’ Cognitive Load during Problem-Solving Activities Based on Electrodermal Activity, Body Temperature, and Heart Rate: Development of a Cognitive Load Tracker for Both Personal and Classroom Use
Automated tracking of physical fitness has sparked a health revolution by allowing individuals to track their own physical activity and health in real time. This concept is beginning to be applied to tracking of cognitive load. It is well known that activity in the brain can be measured through changes in the body’s physiology, but current real-time measures tend to be unimodal and invasive. We therefore propose the concept of a wearable educational fitness (EduFit) tracker. We use machine learning with physiological data to understand how to develop a wearable device that tracks cognitive load accurately in real time. In an initial study, we found that body temperature, skin conductance, and heart rate were able to distinguish between (i) a problem solving activity (high cognitive load), (ii) a leisure activity (moderate cognitive load), and (iii) daydreaming (low cognitive load) with high accuracy in the test dataset. In a second study, we found that these physiological features can be used to predict accurately user-reported mental focus in the test dataset, even when relatively small numbers of training data were used. We explain how these findings inform the development and implementation of a wearable device for temporal tracking and logging a user’s more » learning activities and cognitive load. « less
Authors:
; ; ; ; ; ; ;
Award ID(s):
1742339
Publication Date:
NSF-PAR ID:
10228275
Journal Name:
Sensors
Volume:
20
Issue:
17
Page Range or eLocation-ID:
4833
ISSN:
1424-8220
Sponsoring Org:
National Science Foundation
More Like this
  1. Background With nearly 20% of the US adult population using fitness trackers, there is an increasing focus on how physiological data from these devices can provide actionable insights about workplace performance. However, in-the-wild studies that understand how these metrics correlate with cognitive performance measures across a diverse population are lacking, and claims made by device manufacturers are vague. While there has been extensive research leading to a variety of theories on how physiological measures affect cognitive performance, virtually all such studies have been conducted in highly controlled settings and their validity in the real world is poorly understood. Objective We seek to bridge this gap by evaluating prevailing theories on the effects of a variety of sleep, activity, and heart rate parameters on cognitive performance against data collected in real-world settings. Methods We used a Fitbit Charge 3 and a smartphone app to collect different physiological and neurobehavioral task data, respectively, as part of our 6-week-long in-the-wild study. We collected data from 24 participants across multiple population groups (shift workers, regular workers, and graduate students) on different performance measures (vigilant attention and cognitive throughput). Simultaneously, we used a fitness tracker to unobtrusively obtain physiological measures that could influence these performancemore »measures, including over 900 nights of sleep and over 1 million minutes of heart rate and physical activity metrics. We performed a repeated measures correlation (rrm) analysis to investigate which sleep and physiological markers show association with each performance measure. We also report how our findings relate to existing theories and previous observations from controlled studies. Results Daytime alertness was found to be significantly correlated with total sleep duration on the previous night (rrm=0.17, P<.001) as well as the duration of rapid eye movement (rrm=0.12, P<.001) and light sleep (rrm=0.15, P<.001). Cognitive throughput, by contrast, was not found to be significantly correlated with sleep duration but with sleep timing—a circadian phase shift toward a later sleep time corresponded with lower cognitive throughput on the following day (rrm=–0.13, P<.001). Both measures show circadian variations, but only alertness showed a decline (rrm=–0.1, P<.001) as a result of homeostatic pressure. Both heart rate and physical activity correlate positively with alertness as well as cognitive throughput. Conclusions Our findings reveal that there are significant differences in terms of which sleep-related physiological metrics influence each of the 2 performance measures. This makes the case for more targeted in-the-wild studies investigating how physiological measures from self-tracking data influence, or can be used to predict, specific aspects of cognitive performance.« less
  2. Abstract

    Physical activity can benefit both physical and mental well-being. Different forms of exercise (e.g., aerobic versus anaerobic; running versus walking, swimming, or yoga; high-intensity interval training versus endurance workouts; etc.) impact physical fitness in different ways. For example, running may substantially impact leg and heart strength but only moderately impact arm strength. We hypothesized that the mental benefits of physical activity might be similarly differentiated. We focused specifically on how different intensities of physical activity might relate to different aspects of memory and mental health. To test our hypothesis, we collected (in aggregate) roughly a century’s worth of fitness data. We then asked participants to fill out surveys asking them to self-report on different aspects of their mental health. We also asked participants to engage in a battery of memory tasks that tested their short and long term episodic, semantic, and spatial memory performance. We found that participants with similar physical activity habits and fitness profiles tended to also exhibit similar mental health and task performance profiles. These effects were task-specific in that different physical activity patterns or fitness characteristics varied with different aspects of memory, on different tasks. Taken together, these findings provide foundational work for designing physicalmore »activity interventions that target specific components of cognitive performance and mental health by leveraging low-cost fitness tracking devices.

    « less
  3. Wearable technologies for measuring digital and chemical physiology are pervading the consumer market and hold potential to reliably classify states of relevance to human performance including stress, sleep deprivation, and physical exertion. The ability to efficiently and accurately classify physiological states based on wearable devices is improving. However, the inherent variability of human behavior within and across individuals makes it challenging to predict how identified states influence human performance outcomes of relevance to military operations and other high-stakes domains. We describe a computational modeling approach to address this challenge, seeking to translate user states obtained from a variety of sources including wearable devices into relevant and actionable insights across the cognitive and physical domains. Three status predictors were considered: stress level, sleep status, and extent of physical exertion; these independent variables were used to predict three human performance outcomes: reaction time, executive function, and perceptuo-motor control. The approach provides a complete, conditional probabilistic model of the performance variables given the status predictors. Construction of the model leverages diverse raw data sources to estimate marginal probability density functions for each of six independent and dependent variables of interest using parametric modeling and maximum likelihood estimation. The joint distributions among variables weremore »optimized using an adaptive LASSO approach based on the strength and directionality of conditional relationships (effect sizes) derived from meta-analyses of extant research. The model optimization process converged on solutions that maintain the integrity of the original marginal distributions and the directionality and robustness of conditional relationships. The modeling framework described provides a flexible and extensible solution for human performance prediction, affording efficient expansion with additional independent and dependent variables of interest, ingestion of new raw data, and extension to two- and three-way interactions among independent variables. Continuing work includes model expansion to multiple independent and dependent variables, real-time model stimulation by wearable devices, individualized and small-group prediction, and laboratory and field validation.« less
  4. Driven by the increasing complexity of built environments, firefighters are often exposed to extensive wayfinding information which could cause high cognitive load and ineffective or even dangerous decision making. To reduce injuries and fatal incidents in firefighters’ line of duty, this study aims at measuring the cognitive load and identifying the source of such cognitive overload in wayfinding information review. We developed a Sternberg Test to induce cognitive load on participants pertaining to working memory development, where participants were required to memorize colors, letters, numbers, directions, icons, words, and letter combinations that are relevant to wayfinding tasks. We used an Electroencephalogram (EEG) device to monitor neural activities especially in frontal, parietal, and occipital areas of brain. The fast Fourier transformation (FFT) was applied to separate the sub-band energy. The speed of response in Sternberg Test and the EEG signals were compared to show the coherence between the results of the two methods in representing the cognitive load in the review test. Results indicate that the cognitive load arises from diverse information can be measured to help customize wayfinding information for controlled cognitive load of firefighters in wayfinding tasks.
  5. A large portion of the cost of any software lies in the time spent by developers in understanding a program’s source code before any changes can be undertaken. Measuring program comprehension is not a trivial task. In fact, different studies use self-reported and various psycho-physiological measures as proxies. In this research, we propose a methodology using functional Near Infrared Spectroscopy (fNIRS) and eye tracking devices as an objective measure of program comprehension that allows researchers to conduct studies in environments close to real world settings, at identifier level of granularity. We validate our methodology and apply it to study the impact of lexical, structural, and readability issues on developers’ cognitive load during bug localization tasks. Our study involves 25 undergraduate and graduate students and 21 metrics. Results show that the existence of lexical inconsistencies in the source code significantly increases the cognitive load experienced by participants not only on identifiers involved in the inconsistencies but also throughout the entire code snippet. We did not find statistical evidence that structural inconsistencies increase the average cognitive load that participants experience, however, both types of inconsistencies result in lower performance in terms of time and success rate. Finally, we observe that self-reported taskmore »difficulty, cognitive load, and fixation duration do not correlate and appear to be measuring different aspects of task difficulty.« less