Stress is common and complex. It can benefit people under certain circumstances and increase resilience. Exposure to moderate levels of stress can be beneficial as it can prepare an organism to deal with challenges [13]. On the other hand, stress has also been associated with an increased risk for many somatic and mental illnesses [1], increasing risks for cardiovascular health issues [22] and suppressing the human immune system [23]. Effectively detecting moments of stress in real life may help an individual regulate their stress behaviorally to promote resilience and wellbeing.
Widespread portable devices bring the potential to measure human emotion, including stress levels, using passively sensed data. For example, wearable sensors and smartphones have enabled real-time monitoring of physiological and behavioral data such as body acceleration, skin conductance, skin temperature, heart rate, and phone usage. Prior studies have developed machine learning models to measure momentary self-reported stress levels using physiological, behavioral sensors, and survey features [10,16,37,39]. Although these prior studies provided promising results in stress estimation, we could further improve the performances by addressing challenges in data. In the studies introduced above, the number of self-reported labels is usually limited. On the other hand, wearable devices can collect millions of data samples throughout the study period. The aforementioned studies focused on using data aligned with stress labels, which resulted in information loss when discarding unlabeled data. In alleviating the sparse label issue, semi-supervised algorithms have been widely studied, especially for the computer vision applications [5,12,26,32,41,50,51]. Moreover, inspired by the ideas of these image-based studies, researchers developed methods, including both semi-supervised learning and self-supervised learning algorithms, of leveraging unlabeled wearable data for applications including human activities recognition [21,36,45,56], cardiovascular risk prediction [3], and stress detection [38,49].
The aforementioned studies achieved success in their respective fields, and they all illustrated that the information from unlabeled data could boost the performance for the supervised tasks. However, there are issues needed to be resolved when utilizing these methods on wearable data from stress detection because of the inconsistency of the domain challenges and knowledge. First, the labeled and unlabeled data collected in the wild can be in different data distributions, which breaks a critical prerequisite of the aforementioned methods in leveraging unlabeled data for helping the supervised tasks. For example, massive amounts of sleep time samples exist in the databases, whereas the labeled data are collected during non-sleep periods. Secondly, as an essential tool of the methods mentioned above, the data augmentation techniques can be under-studied in the wearable data domain, and different augmentations contribute differently to the robustness of the model [52]. Therefore, in this work, we proposed a two-stage semi-supervised learning framework with modules to alleviate the above-mentioned issues. We designed an active sampling method to explicitly select unlabeled samples that obey the same distribution as the labeled data. We also introduced the averaging mechanism with a consistency regularization method to alleviate the detriment of the learning parameters from improper augmentations. Our contribution can be summarized as follows:
• We developed a two-stage semi-supervised learning framework that includes an auto-encoder pre-training method and consistency regularization to leverage both labeled and unlabeled data for robust model training.
• We propose an active sampling approach for selecting unlabeled data in semi-supervised learning to reduce the distribution differences between the labeled and unlabeled data for the stress detection system. • We evaluated the proposed methods using two datasets, including multimodal sensor data and momentary stress labels collected in the wild. Additionally, we conducted an ablation study on four human activity recognition (HAR) datasets. We observed clear improvements in model performance using the proposed methods compared to the baselines. • We conducted empirical experiments to examine state-of-the-art (SOTA) methods from other domains in wearable-based stress detection tasks. Our proposed method showed competitive performance compared to the SOTA methods.
Traditionally, stress has been measured using surveys. For example, the Perceived Stress Scale (PSS) was developed for measuring perceived stress over the past month [8]. The Holmes and Rahe Stress Scale is another survey instrument that adds up the self-reported events in the prior year that could lead to stress, estimates the total amount of stress in the past year, and determines whether people are at risk of becoming sick [19]. However, a drawback of these traditional surveys is that they might be cumbersome for individuals to complete, especially when the studies last for a long time. The resulting fatigue may make survey answers less reliable.
With the development of mobile phones and wearable devices, accessing users' physiological and behavioral data in daily life settings has become a boost in monitoring human mental status. Machine learning has enabled us to develop models to learn patterns from data samples and has already brought benefits to ubiquitous computing applications. Multi-modal data from wearable sensors, mobile phones, and other smart devices have been widely used with machine learning in estimating human emotion as well as momentary stress levels [4,10,16,18,24,27,37,39,54]. [39]. Each subject in the study was exposed to a protocol, including four stressors and six rest periods, and stress labels were collected before and after each stressor/rest period through interviews. The authors proposed a personalized SVM algorithm to classify binary stress labels (low/high), which provided a precision of 0.68 with a recall of 0.80. In these studies, while physiological data were collected continuously using wearable sensors, human stress labels were collected using questionnaires. To match the sensor data with sparse labels, the collected physiological data were downsampled to align momentary features with stress labels, which [37,39] caused a loss of information.
To overcome the difficulty in learning models with a small number of labels, numerous semi-supervised learning methods have been developed to leverage massive unlabeled samples [5,12,26,32,41,50,51]. For example, Laine et al. proposed an Π-model to infer the predictions from two transformed images from a single unlabeled source and regularized the discrepancy between two output heads [26]. Berthelot et al. proposed a MixMatch framework that integrated multiple components to improve the model performance using unlabeled data. They first utilized the mix-up approach [57] to transform original unlabeled samples into 𝐾 augmented versions and employed a sharpening algorithm to minimize the entropy of predictions.
Leveraging Unlabeled Wearable Data. Semi-supervised learning methods have also been applied in mobile and wearable device studies [3,31,42], where models learned representations from massive unlabeled data and used a small amount of labeled data to train the supervised prediction models. Ballinger et al. applied a semi-supervised auto-encoder pre-training method [11] and used physiological data collected from wrist-wearable devices to detect symptoms of cardiovascular disease. Their semi-supervised model improved AUC scores for three out of four types of symptoms, and the highest improvement rate was 10.5% in detecting high cholesterol.
In momentary stress detection, Maxhuni et al. used a self-training tree model in combining unlabeled wearable sensor-based physiological data with labeled data [31]. Compared to the supervised learning method, their binary stress detection performance in the F1 score boosted from 66.0% to 70.0%. Besides the SOTA works in computer vision, leveraging unlabeled samples with contrastive constraints and consistency regularization have also been examined on wearable sensor data for human activities recognition (HAR) tasks [2,21,28,36,44,45] and emotion classification [38,49] learned representations to supervised learning with labeled data. Similarly, Chi et al. pre-trained an encoder by classifying the data augmentations using unlabeled data and transferred the learned representations for HAR [45]. Liu et al. proposed a semi-supervised SimCLR [9]-based framework named SemiC-HAR to leverage the unlabeled data for HAR. [28] Jain et al. designed a contrastive constraint on the different temporal positions of unlabeled data collected from a multi-sensor accelerometer system. These methods achieved state-of-theart performances in multiple applications with the prerequisite that the labeled and unlabeled data are in the same distribution space. However, for in-the-wild stress detection tasks, unlabeled samples can be in different distributions from the labeled samples even when those samples were collected from the same subjects. Besides, data augmentation methods were not guaranteed to help the model learn robust parameters. Thus, in this work, we propose to improve these methods by actively selecting unlabeled samples based on the latent distribution for semi-supervised learning. Also, we designed an averaged consistency regularization method in case of the improper noise introduced by augmentations.
This section introduces our proposed semi-supervised learning method for leveraging all the data 𝑋 , including both labeled 𝑋 𝑙 and unlabeled sequences 𝑋 𝑢 in detection stress labels 𝑦. A two-stage semi-supervised learning framework is proposed in this section. Moreover, we introduce a domain knowledge-based pseudo-annotating method to enrich the training data.
Figure 1 shows the overall framework of the proposed semi-supervised learning method. Using the actively sampled unlabeled data, the encoder, which is employed as the feature extractor, is pre-trained in an unsupervised manner with an auto-encoder structure. Further, the fine-tuning process involves both the labeled and unlabeled data. The consistency regularization with sequential data augmentation approaches is introduced to improve the robustness of the model. Following countless proven studies on modeling sequential physiological data [35], we use a one-dimension convolutional neural network (1D CNN) to extract information directly from the raw Fig. 2. Examples of data augmentation on a sequential sample data. Thus, there is no feature extraction procedure required in this study. Additionally, unlike previous relevant HAR frameworks, such as SemiC-HAR, our framework does not incorporate pre-trained supervised model-based pseudo-label annotation, due to the challenging nature of the stress detection task. Instead, besides the basic two-stage proposed framework, we propose a novel unsupervised representation-based active sampling method for selecting unlabeled samples, which is introduced in Section 3.1.3.
Unsupervised Pre-training with Auto-encoder. Typically, a supervised learning inference can be equalized in a close form of:
𝑓 is an encoder for extracting the features from signal 𝑋 , and ℎ is the classifier for predicting output ŷ. Without the stress labels, the optimization of ℎ becomes difficult as lacking supervision for ŷ. However, learning robust parameters in the encoder 𝑓 is feasible with the information contained in the unlabeled physiological data. In this study, we applied the 1D CNN-based structures to pre-train the model through the unlabeled samples. The structure of an auto-encoder can seem as follows:
Same as the supervised learning structure, 𝑓 represents the encoder. With the extracted representations from 𝑋 𝑢 , the auto-encoder aims to reconstruct the input signal 𝑋 𝑢 as X𝑢 by decoder 𝑔 on top of the learned features 𝑓 (𝑥 𝑢 ). Thus, the objective function of the auto-encoder can be defined as the mean-square-error (MSE) loss between 𝑋 𝑢 and X𝑢 :
By optimizing the L 𝑎𝑒 , the encoder learns the weights of extracting informative features that represent the input signal. After training the auto-encoder, the parameters learned in the 𝑓 are transferred to the model of stage 2 as the initialization of the encoder.
Semi-supervised Fine-tuning with Consistency Regularization. Consistency training methods regularize model predictions to be invariant to slight noise applied to input [32,51]. The theoretical foundation of this method is that a robust machine learning model should be able to tolerate any slight noise in an input example. For example, when inputting a data sequence and its augmented sequence into a robust model, the outputs of those two input examples should be the same. Since there were plenty of unlabeled sequences in our datasets, ideally, the concept of consistency regularization could bring robustness to our model. In our study, inspired by [51], we conducted consistency training combined with the augmented data for time-series data. As illustrated in [52], the effectiveness of data augmentation on sequential data is affected by multiple factors. Thus, we designed a multi-head consistency regularization to avoid the risk of learning ill-state representations with inappropriate augmentations. We randomly generated 𝑀 augmented sequences using labeled/unlabeled samples in the same training batches. The consistency loss aims to regularize the similarity of predictions from the original data and the augmented samples. For example, since our task was to estimate stress status in binary classification, the supervised loss was designed as a cross-entropy loss. We applied the Kullback-Leibler divergence loss as our designed consistency loss. To present the method in the formula, the final objective function of training stage 2 with the consistency regularization method is:
where X𝑙 is the augmented labeled sequence, and X𝑢 is the augmented unlabeled sequence. The probability 𝑝 (𝑦|𝑥) indicates the likelihood of getting model results with given data 𝑥. In our case of classification, 𝑝 (𝑦|𝑥) is the sigmoid output for binary classification. 𝛼 controls the weights of the consistency regularization. The supervised consistency regularization coefficient 𝛼 is set with a ramping-up function 𝑤 (𝑡) to avoid noisy distortion in the early training stage. Data Augmentation. To perform the consistency regularization, we adopted four types of data augmentation techniques for time-series data from [47], including jittering, scaling, time warping, and magnitude warping. Jittering (J) adds tiny Gaussian noise to the original signals. For scaling (S), the original signals are scaled by generated Gaussian random numbers (N ∼ (1, 0.05)). Time warping (TW) perturbs the temporal characteristics of the data. The temporal locations of the samples are changed by smoothly distorting the time intervals between samples. Magnitude warping (MW) changes the magnitude of each sample by convoluting the data window with a smooth curve varying around one with a standard deviation of 0.05 (N ∼ (1, 0.05)). The essence of these methods is adding a small amount of noise to time-series data so that the trained model will be robust. Figure 2 shows an example of different DA methods on a sequence of electrodermal activity data. The green lines are the original signal, and the red lines represent the data generated using four different DA methods.
Pseudo Labels
Fig. 4. The stress pseudo annotation approach. The equal value labels are assigned to the neighbor time points within symmetric windows of the self-reported time point.
Wearable physiological signals collected in the wild contain noise. In addition, human physiological signals can vary widely in different states. These noises and uncertainty can cause significant distribution differences between labeled and unlabeled data samples. The distribution of the unlabeled data is usually more comprehensive, and a significant fraction of them might even distribute differently from the labeled data.
To reduce the influence of noise and unlabeled samples with different distributions to the model, we propose an active unlabeled sample selection method. We first trained an auto-encoder structure as in 3.1.1 with labeled data, then clustered all labeled samples in latent space low-dimension representation using a Gaussian mixture model (GMM). After analyzing the elbow points of both the Akaike and Bayesian information criteria, we fixed the number of Gaussian components as K. Then, we used the trained encoder to infer the latent representations of all the unlabeled samples as 𝑓 (𝑥 𝑢 ). The negative log-likelihood of each unlabeled sample, which is the probability of the observed data under the trained GMM model, can be calculated via the following equation:
where 𝛾 represents the weight mixture component, 𝜇 and Σ are the learned mean value and co-variance of the corresponding Gaussian component. Then, we selected the unlabeled samples that most possibly obey the similar distributions of labeled samples based on the calculated negative log-likelihood values (NLL). The smaller the NLL, the more similar the sample distributed as labeled data. Figure 3 shows the reduced-dimensional visualization of latent space representations, and the red dash line indicates the example of sampling boundary by negative log-likelihood levels across the whole dataset. Under this scenario, we only focus on the unlabeled data within the red boundary in the semi-supervised learning framework.
Based on the fact that stress in human daily life may not change rapidly, stress status usually takes minutes to be relieved [20]. While in real-world studies, self-reported stress annotation only represents the subjective stress status of study participants at a certain time point. To enrich the number of labels with the slow-changing attribute of human stress, we applied a time window-based pseudo annotating strategy as illustrated in Figure 4. A time window 𝑡 was applied before and after the time point where study subjects reported their stress status 𝑦, and the time points within 𝑡 are annotated with the same label value of 𝑦. The window length 𝑡 was treated as a hyperparameter that requires fine-tuning while training.
This section introduces experimental settings such as datasets, model structures, training hyperparameters, etc. The baseline models and the reproduced SOTA methods are also described. Then, the evaluation results of the proposed method are listed for various methods and datasets.
We describe two datasets we used to evaluate our methods. The meta-information of the datasets can be found in Table 1.
4.1.1 Dataset I: SMILE. Wearable sensor and self-report data were collected from 45 healthy participants (39 females and 6 males) for 390 days. The average age of participants was 24.5 years old, with a standard deviation of 3.0 years. Participants contributed to an average of 8.7 days of data, with a minimum of 5 days and a maximum of 9 days. Two types of wearable sensors were used for data collection [40]. One was a wrist-worn device (Chillband, IMEC, Belgium) designed for the measurement of skin conductance (SC), ST, and acceleration data (ACC). The SC was sampled at 256 Hz, ST at 1 Hz, and ACC at 32 Hz. Participants wore the sensor for the entire testing period but could take it off during the night and while taking a shower or during vigorous activities. The second sensor was a chest patch (Health Patch, IMEC, Belgium) to measure ECG and ACC. It contains a sensor node designed to monitor ECG at 256 Hz and ACC at 32 Hz continuously throughout the study period. Participants could remove the patch while showering or before doing intense exercises.
In addition to the physiological data collected by sensors, participants received notifications on their mobile phones to report their momentary stress levels 10 times per day, spaced out roughly 90 minutes apart for eight consecutive days. In total, 2494 stress labels were collected across all participants (80% compliance). The stress scale ranged from 1 ("not at all") to 7 ("Extreme"). In 45% of the cases, participants reported that they were not under stress, while in only 2% of the cases did they report that they were under extreme stress.
Data Processing: In this work, we focused on the physiological signals of ECG and GSR in modeling stress status. We preprocessed the ECG data with a high-pass Butterworth filter with a cutoff frequency of 0.5 Hz and an order number of 5. Also, a powerline filtering for the white noise at 50Hz was applied after the Butterworth filter. For the raw GSR data in the SMILE data, we decomposed the signal into phasic and tonic components, and only the tonic components were used as the model input in order to preserve the long-term temporal information and avoid introducing noises to the model. The implementation of the aforementioned procedures was based on the NeuroKit2 Python library [29]. The ECG and GSR signals were segmented into one-minute windows and aligned based on the timestamps. Around 480,000 bimodal sequential inputs were constructed with 2494 annotations. Further, for the stress labels, we binarized the stress levels by categorizing stress level 1 as a class of "non-stressed" (45%) and levels 2-7 as the "stressed" class (55%).
4.1.2 Dataset II: TILES. Tracking Individual Performance with Sensors (TILES) is a multi-modal data set for the analysis of stress, task performance, behavior, and other factors to professionals engaged in a high-stress workplace environment [33]. The dataset was collected from 212 participants for 10 weeks. In this work, we leveraged the ECG signals collected by the chest-worn OMSignal smart garments, which were not collected in a strictly continuous manner. At 5-minute intervals, the sensor collected ECG signals for fifteen seconds at a sampling rate of 250 Hz for the participants. Regarding the stress labels, participants annotated stress levels through multiple 5-point scale questions.
Note that prior research has been conducted on this dataset. For example, Gaballah et al. leveraged TILES audio and physiological data with a bidirectional LSTM network and inferred stress labels in a binary classification task with an F1-score of 0.64 [14].
Data Processing: Since our interest lies in leveraging physiological data collected from wearable sensors, only the ECG data were processed and utilized. Similar to the preprocessing procedures for ECG for the SMILE dataset, we applied a high-pass Butterworth filter with a cutoff frequency of 0.5 Hz and an order number of 5; also, a 50 Hz powerline filter was applied for filtering out the white noise. However, since the ECG was not collected continuously in the TILES dataset, we used 15-second data sequences every 5 minutes, and the alignment of ECG and stress annotations are also in a resolution of 5 minutes. Following the stress label processing procedures in [14], We calculated the z-scores of stress levels for each individual, considering the subjective variability, and then divided them into two classes, class 0 (non-stressed, z-score below the average) and class 1 (stressed, z-score above the average). Overall, around 370,000 sequential samples were processed with 600 stressed labels and 629 non-stressed labels.
To fairly leverage the data from subjects, we applied a subject-independent cross-validation setting in training and testing sets split. Five folds are split evenly for both SMILE and TILES datasets. For example, for the SMILE dataset, on each split, data from 36 subjects are used as the training set, and the rest data is employed as the testing set for performance evaluation.
To ensure the reproductivity of the proposed method, we describe the model structures and hyperparameters of the proposed method and the reproduced semi-supervised learning SOTA methods.
To implement the proposed structure, we utilized the model structures including, ECG encoder, ECG decoder, GSR encoder, GSR encoder, and classifier. ECG Encoder & Decoder. We used the same 1D CNN structure in extracting information from ECG sequences for both SMILE and TILES data as the sampling frequencies of measuring ECG are close. The ECG encoder structure consists of 5 layers of CNN layers with kernel sizes of [8,6,5,3,3] and samples the channels of signal from 1 into [16,32,64,128,256] respectively. We used an average pooling layer at the output of the encoder and obtain vectors with a length of 256 for the classifier.
The structure of the ECG decoder was symmetric as that of the ECG encoder. Totally 5 layers of the 1D transposed convolutional layers (1D TranCNN) function to reconstruct the input signal directly from the output of the encoder. Thus, the kernel size and the out-channel number were [3,3,5,6,8] and [128, 64, 32, 16, 1], respectively. GSR Encoder & Decoder. The GSR encoder and decoder were used to extract latent representations for the SMILE dataset. The encoder structure contained 3 layers of CNN layers with kernel sizes and out-channel dimensions of [8,5,3] and [16,32,64], respectively. Symmetrically, the kernel sizes and the out-channel dimensions for the TranCNN layers of the decoder structures were [3,5,8] and [32,16,1], respectively. Classifier. As the design of the encoder structures, we obtained features in dimensions 256 and 64 for ECG and GSR, respectively. Thus, the input dimension of the classifier for the SMILE dataset was 320; and the dimension of the TILES dataset was 256. One embedding layer of 512 dimensions connected the input layers for both datasets, and an output layer for class number 2 is employed.
Reproduced SOTA Semi-supervised Learning Methods. To conduct an empirical comparison between our proposed method and the SOTA methods on the wearable-based stress detection task, we reproduced the following methods that have been proven in computer vision tasks: two consistency regularization-based SOTA methods such as Π-model [26] and virtual adversarial training (VAT) [32] and two hybrid semi-supervised learning methods including interpolation consistency training (ICT) [50], MixMatch [5], and FixMatch [41]. Also, we conducted experiments on the proven methods of leveraging unlabeled samples on wearable data for HAR, e.g., SemiC-HAR [28] and SelfHAR [45], as reproduced SOTAs for comparison. See the details about the SOTA semi-supervised learning methods below. Π-model [26]: The Π-model operated two different transformation for an unlabeled input 𝑥 𝑢 , to form 𝑥 ′ 𝑢 and 𝑥 ′′ 𝑢 so that the model predicted 𝑦 ′ 𝑢 and 𝑦 ′′ 𝑢 . Then the model constrains the consistency of the two results. We implemented Π-model with two different DA approaches in section 3. VAT [32]:This algorithm constrained the consistency of a signal and its transformation with additive noise, the trainable adversarial perturbation 𝑟 . The perturbation 𝑟 was trainable, which was constrained by coefficient 𝜉 = 1 × 10 -6 to avoid gradients explosion in our implementation. We allowed 5 iterations for each sample in a single epoch to update the parameter of 𝑟 . Based on the model performance in model training, we adjusted to ramping up epoch in equation ( 5) as 30 with a coefficient 𝑐 of 0.3. ICT [50]: The ICT algorithm used the mix-up method, which summed the original unlabeled data to generate the augmented samples. In our implementation, the mix-up coefficient was set as 0.2, which means we summed up 0.8 • 𝑥 1 𝑢 and 0.2 • 𝑥 2 𝑢 as a new signal as 𝑥 ′ 𝑢 . Then, the model optimized the discrepancy between the prediction 𝑦 ′ 𝑢 and {0.8 • 𝑦 1 𝑢 + 0.2 • 𝑦 2 𝑢 }. Also, we reproduced the average teacher strategy [46] with an updating factor of 0.999. We adjusted the ramping up epoch in equation ( 5) as 20 with a coefficient 𝑐 of 80. MixMatch [5]: The MixMatch approach combined multiple prior techniques, such as consistency regularization, entropy minimization, and mix-up DA approach, to serve as a semi-supervised learning framework. Similarly to ICT, we also reproduced the mix-up approach in the MixMatch algorithm with a mix-up coefficient of 0.2. In the steps of sharpening prediction and reducing model entropy, we set the averaging bag size as 3 for each sample with a normalizing temperature of 0.5. The ramping-up epoch in equation ( 5) was set to 30 with a coefficient 𝑐 of 100. FixMatch [41]: The FixMatch approach relied on the consistency of the outputs from the weakly-augmented samples and strongly-augmented samples. We set the jittering and scaling methods as the weak augmentations; while the TimeW and MagW as the strong augmentations. Based on the model confidence from the weaklyaugmented predictions, pseudo labels were annotated to enrich the training set. We set the threshold of pseudo labeling as a threshold of 0.95 to the model output after softmax. The ramping-up epoch in equation ( 5) was set to 20 with a coefficient 𝑐 of 80. SelfHAR [45]: The SelfHAR is a teacher-student-based representation learning that pre-trains the model encoder with a pre-task of recognizing various data transformations applied to the original signal. We reproduced this approach based on the data augmentation methods we introduced in section 3.1.2, and the unlabeled samples were selected from the teacher model with a confidence threshold of 0.9. With 100 epochs of pre-training, the accuracy rate of distinguishing augmentation methods achieved over 98%. Then, the pre-trained weights in the encoder served as the initial encoder weights in supervised learning. We fine-tuned the model for the stress detection task in 50 epochs with a learning rate of 1e-3.
The SemiC-HAR presents a 4-stage framework that involves supervised training, self-labeling, contrastive learning with selected samples, and fine-tuning. The framework used the labeled data to train a supervised learning model and annotate the unlabeled samples with the model logits as the likelihood scores from the trained supervised model. Then, the framework pre-trained the encoder using SimCLR [9] with samples selected by the prediction confidence, which was set to 0.9 following the original study. Finally, the pre-trained encoder was fine-tuned with our stress prediction task in 50 epochs with a learning rate of 1e-3.
In the performance evaluation section, we aim to answer the following research questions on the datasets introduced in section 4.1:
• Q1. What is the baseline performance of stress prediction in the wild?
• Q2. Do all the semi-supervised learning components contribute to the performance? • Q3. Is active sampling unlabeled data helpful in stress detection?
• Q4. How do well-proven semi-supervised learning methods from other domains work in stress detection?
To understand Q1, we designed two baseline approaches including a random guessing baseline, a major class baseline, and a purely supervised 1D CNN baseline. Random baseline is the method that assigns labels to test instances according to the class probabilities in the training set [6]. For example, in the SMILE dataset, the probability of class 0 𝑝 (𝑦 = 0) = 0.45, we assigned the instances in the test set as class 0 with the probability of 0.45. Major class baseline means we predict all the samples from the evaluation dataset to be in the major class, e.g., predicting all "stressed" for the SMILE dataset and all "non-stressed" for the TILES dataset. Besides, we examined the model performances of using pseudo labels proposed in section 3.2 along with the baseline methods, as this method is straightforward and intuitive. The tuning of the time-window length is discussed with the ablation studies as section 6.1.
We answer Q2 by experiments utilizing the semi-supervised methods, including auto-encoder pre-training and consistency regularization, in individual and combined manners. Further, by testing the performances of whether conducting the active sampling unlabeled samples, we explore the question Q3. Moreover, we examined and compared the performances of the reproduced methods in section 4.2.2 to answer Q4. Note that, the experiments in Q2, Q3, and Q4 are based on the training set that has been expanded by pseudo labels.
Table 2 shows the results of evaluations in the average accuracy rate and macro scores with standard deviations. According to model performances, we summarize our results as follows: Q1 (Baseline stress prediction performance): According to the performance comparison between the random baseline and the 1D CNN baseline, we found that stress detection in the wild is intuitively challenging. Even though statistical differences (paired t-test, p < 0.05) were observed on both datasets for both accuracy and macro F1 scores, the supervised learning method output only outperformed the random guessing by a small margin. Also, we found that applying the time windows-based pseudo labels in supervised learning improved the model performance significantly (paired t-test, p < 0.05). Q2 (Contributions of Semi-supervised learning): In Table 2, we listed the performance of using the semisupervised framework on top of the pseudo labels. For both auto-encoder pre-training and consistency learning approaches, applying them individually or together boosted the model performance compared to using pseudo labels only by clear margins. Also, the results suggested the consistency regularization method achieved higher performances than the auto-encoder pre-training method; while the combination of both methods provided the best performances on both datasets (ANOVA, Tukey, p < 0.05.).
The last 3 rows in Table 2 show the performance of semi-supervised learning approaches with active sampled unlabeled data. Compared to the results without applying the active sampling techniques, active sampling significantly improved the model performance for both auto-encoder pre-training and consistency regularization on both the SMILE and TILES datasets (paired t-test, p < 0.05). Nevertheless, we did not observe statistical differences when comparing the results of using both semi-supervised methods on the SMILE dataset even with higher averaged performances (paired t-test, p > 0.05). Q4 (Comparison with SOTA semi-supervised learning methods): The performances from 5 reproductions of well-proven semi-supervised methods all achieved improvements on top of the pseudo-labeling method. On the SMILE dataset, FixMatch achieved the best performance in both accuracy and F1 score; whereas VAT performed the best on TILES. However, ICT and MixMatch, which were developed on top of the mix-up augmentation approach showed lower margins of improvement compared to other methods. When comparing these reproduced methods with our proposed framework, our method showed statistically high macro scores than those with and without an active sampling approach applied (ANOVA, Tukey, p < 0.05).
This section covers the ablation studies of essential hyperparameter tuning in this study.
As shown in the results section, enriching the training set can help improve the test performance in stress detection. However, the proper length of the window of applying pseudo labels is the essential hyperparameter to determine. Larger window sizes inject more samples into the training set while introducing the risk of biased samples at the same time. This section shows experiments with time window tuning on both the SMILE and TILES datasets. Figure 5 shows the performances of applying different time windows in generating pseudo labels.
For both the SMILE and TILES datasets, we observed the trends that the model performance first increased and then decreased with longer window sizes. Based on these results, we fixed the window sizes to be 10 and 15 minutes for the SMILE and TILES datasets, respectively.
We propose an active sampling method for selecting unlabeled in this study. To conduct the most effective and efficient training process, we may select the least amount of data to get the highest performance as the ideal cases. Further, since our proposed method relied on the cluster of the latent representations, we also explore another clustering algorithm, k-Means (k=6), to compare with the GMM method proposed in section 3.1.3. Moreover, to verify the effectiveness of active unlabeled sample selection, we also evaluated the pre-trained models with randomly sampled data. We tune the volume of the selected unlabeled samples by conducting experiments on both datasets in the auto-encoder pre-training task. Figure 6 shows the performances of using different portions of the unlabeled samples while tuning. For both SMILE and TILES data, the active sampling method outperformed the random sampling baseline. Also, with active sampled unlabeled data, the semi-supervised learning algorithm achieved the highest performances without leveraging all the unlabeled samples. In addition, based on our ablation studies, we selected GMM as the clustering algorithm instead of the kMeans as GMM provided higher macro F1-scores with fewer unlabeled samples. Based on the results, we selected 40% of the unlabeled samples for both the SMILE and TILES datasets using GMM.
As introduced in section 4.1, the datasets of both SMILE and TILES contain multiple physiological modalities.
Besides the modalities we leveraged in the experiments, ST and ACC are available in SMILE, and minute-to-minute features such as heart rate, step count, and sleep status are available with the Fitbit wristband in the TILES dataset. Considering the complexity of the model, we desire to achieve the best results with the least amount of modalities. Thus, we conducted ablation experiments on the selection of modalities. For example, Table 3 Fig. 6. Model performances versus the volumes of the selected unlabeled samples for SMILE (upper) and TILES (lower) datasets. Auto-encoder pre-training method was used in tuning the selected volumes. And methods including GMM, kMeans, and random sampling were compared. The * marker indicates the volume of the unlabeled samples we selected for the other experiments in this study.
shows the average model performance of supervised learning 1D-CNN on the SMILE dataset using different combinations of modalities. From the table, we found the supervised model did not learn useful information in detecting stress using modalities such as ST and ACC. The combination of ECG and GSR modalities provided the best results. Our results are consistent with previous findings in the literature. For instance, GSR is a tonic and phasic electric signature on the skin that changes when a person experiences stress or emotional arousal as a result of sweat gland activation [7]. ECG reflects human heart activity, including heart rate variability, which decreases under stress due to sympathetic nervous system activation [48]. Researchers have also demonstrated that utilizing and combining multiple sensor data in multimodal methods, enabled by these physiological signals, can provide additional benefits [15,17,55].
On the TILES dataset, we examined the supervised 1D-CNN performance with combinations of (1) ECG, (2) Fitbit features, and (3) ECG + Fitbit features. We observed the supervised performances for these three combinations to be 50.79 (3.21), 48.52 (3.45), and 50.90 (3.27), respectively. Although the combination of ECG and Fitbit showed slightly higher average performance than using ECG only. We did not find any significant
We proposed and evaluated a semi-supervised learning framework in stress detection; however, auto-encoder pre-training and consistency regularization are task-agnostic. Thus, as an ablation study, we tested the capability of the proposed method in the HAR tasks. Following the settings in [28,45], we applied our method to four HAR datasets including the Motion Sense [30], HHAR [43], UCI HAR, and WISDM [25] datasets, which contain motion-related sensing data collected from portable devices and labels of different activities. The raw datasets are pre-processed using the released source codes in [45], which generates pairs of sequential samples and activity labels. To construct the sparsely annotated datasets, we followed the same strategy as in [28], which was to mask out 90% of the samples as unlabeled samples and conduct the experiments on the rest 10% of the samples. We randomly selected 10% of the labels in this ablation study. The weighted F1 score was used as the evaluation metric. Table 4 shows the evaluation results of the conducted HAR ablation study. We observed that compared to the supervised model, our method improved the performance using the unlabeled samples. Also, our method achieved competitive performance compared to SOTAs. Note that in the HAR experiments, the proposed active sampling method did not contribute to the unlabeled sample selection, as the distribution shifts between the labeled and unlabeled samples were not significant. In addition, we did not annotate the pseudo labels as in section 3.2 as activities can change more rapidly than stress levels.
To further evaluate the cross-dataset robustness of the proposed method, we also explored the possibility of merging datasets with similar signals to enlarge the volumes of available annotations. For example, both the SMILE and TILES datasets contain in-wild stress labels and ECG signals. We conducted an ablation study of cross-dataset evaluation where we trained a model on the SMILE dataset but tested it on the TILES dataset and vice-versa, to examine the cross-dataset robustness. To adjust ECG signals in both datasets, we resampled the ECG signal from the SMILE dataset to 250 Hz, and truncated the samples in the SMILE dataset to a length of 15 seconds. To simplify the evaluation, we focused on two settings: (1) the supervised learning approach with original and pseudo labels, and (2) the semi-supervised learning approach with active sampling, auto-encoder pre-training, and consistency regularization. Table 5 shows the cross-dataset evaluation results in average macro F1 scores. From the table, we can observe substantial performance drops across different datasets. Multiple factors might result in these performance drops. For example, ECG data were collected using different devices (chest patch vs smart garment) and the location of measurement on the chest might be slightly different. Different scales were used to collect self-reported stress labels. Figure 7 shows the visualization of the latent space from SMILE and TILES datasets, where we can observe that the representations do not obey the same distribution.
The results of this ablation study showed that merging different stress datasets can be challenging and requires additional approaches to tackle distribution drift issues.
In this section, we discuss semi-supervised learning algorithms in stress detection. Our results showed that the mix-up augmentation method, which has been well-proven in the other domain, did not perform well on our evaluating databases. Therefore, we discuss the insight into applying a mix-up approach for wearable data. Besides, we compare this work with the other works that utilized the same databases in the literature. Moreover, this section covers the biased label challenge we found during the experiments. Lastly, we conclude our discussion by summarizing the implication and limitations of this study.
The mix-up augmentation approach is well-applied in the computer vision domain and achieves promising results in semi-supervised learning studies. However, according to the results of the reproduced SOTA in section 4.3, we found the mix-up method did not benefit the semi-supervised learning in our datasets as observed in the computer vision domain. We believe the misalignment of the temporal information invalidated the mix-up augmentation. Figure 8 shows an example of mixing two ECG sequences. We observed that the mix-up method can generate excessive distortion of the ECG signal. For example, without alignment in the temporal positions, the difference in the signal amplitude between the peak and plateau can easily create pulses in the plateau region or distort the waveform at critical positions (e.g., QRS areas in ECG).
Several studies have been conducted for human momentary stress detection using the same databases. Even though there are differences in experimental settings, we compare our experiments and results with the studies in the literature. Since the raw data in SMILE are not publicly accessible at this moment, we focus on the comparison with the literature on TILES in this section. Table 6 shows the comparison among the studies for stress detection tasks with the TILES dataset. Gaballah et al. extracted crafted features from audio, locations, and wearable sensors in 30-minute windows for consecutive 48 hours before the stress annotated time point [14]. They applied a bidirectional LSTM network to model the extracted feature sequences into stress status. Pimentel et al. [34] and Yang et al. [53] engineered features from wearable data, including ECG signal and features collected by Fitbit. Pimentel et al. [34] extracted high-level heart rate variability (HRV) features across a time window of 24 hours and predicted stress status with an SVM [53] leveraged HRV and Fitbit features of every 5 minutes with a total of 120 minutes window (5 minutes × 24 time steps), and they developed the LSTM method to estimate the stress status for subjects. According to the evaluation results shown in the table, we found that leveraging the information in longer data sequences may contribute to the model performance in accuracy. Also, as indicated in [14,52], multi-modal learning helped improve the performances in stress detection on the TILES dataset. Our method showed a similar level of performance in accuracy rate compared to [34,53], with the information from remarkably shorted data sequences (15 seconds) of ECG data only.
The stress labels are annotated by subjective self-reported questionnaires reported by study participants during the study period. Thus, it is highly possible that biases are introduced into the stress labels. Moreover, due to the participant-wise heterogeneity in stress perception, the stress labels can be biased across subjects when considering a generalized model. The biases in labels are reflected in the test accuracy. Figure 9 shows the model performance in accuracy for each individual stress status on the SMILE dataset. As mentioned in section 4.1, study participants reported their stress status in 7 different levels (1 -7), and we binarized the stress labels by 1 vs. 2-7. From and figure, as well as according to the statistical test (ANOVA. Tukey, p < 0.05), we can observe lower performances in accuracy when the original stress status was reported as 2 and 3 compared to the original status ≥ 5. This phenomenon suggests that the model has difficulty distinguishing the features embodied by the physiological signal when the stress level is at the boundary point of whether or not the participant was stressed. In other words, the stress labels around the boundary point can be "biased" to the model as these samples were more challenged to be correctly classified by models. To avoid biases from these uncertain annotations, we propose discarding the labels in the boundary or using a different label strategy while binarizing the stress labels. For example, in our case, we could try non-stress vs. stressed in the original status of 1 vs. 4-7 on the SMILE dataset. and Human Activity Recognition (MotionSense) tasks. Using the same selection thresholds, the accuracy of pseudo labels in the HAR study is substantially higher than that of the stress detection task.
We conducted a thorough evaluation of our proposed framework's performance on stress detection tasks. Additionally, we examined the model's performance on the HAR tasks, comparing our method with strong baselines such as SelfHAR and SemiC-HAR in our ablation studies. Interestingly, we found that the HAR-originated SemiC-HAR method performed competitively in HAR tasks, but exhibited weaker performance in stress detection tasks. This phenomenon might be related to the lack of robustness of the pseudo-annotation stage of SemiC-HAR, which relies on a pre-trained supervised classifier using model likelihood confidence scores as mentioned in Section 4.2.2.
In contrast to the HAR task, in-wild stress detection is more challenging, resulting in high incorrect annotation rates during the supervised learning-based pseudo-annotation step. For example, Figure 10 illustrates the relationship between the accuracy of supervised based pseudo labels and model likelihood confidence scores on the SMILE and Motion Sense datasets, respectively. The supervised accuracy on the Motion Sense dataset is substantially higher than the performance on the SMILE dataset. Furthermore, the supervised learning performance on the SMILE dataset suggests that even with a high selection threshold, the pseudo-labels based on supervised learning are noisier than those in the Motion Sense dataset. For instance, when using a threshold of 0.90, the accuracy in pseudo annotation on the Motion Sense dataset can be higher than 95%; whereas the accuracy on the SMILE dataset is lower than 60%. In addition, as revealed in Section 6.4, we did not observe a significant difference in the latent space representations between the labeled and unlabeled datasets in HAR tasks. Consequently, our unsupervised active sampling method did not fully function when performing the HAR tasks. Thus, although there are similarities, such as unsupervised pre-training and consistency regularization, between the methods, our proposed method outperforms the HAR-originated methods including SelfHAR and SemiC-HAR in stress detection.
Semi-supervised learning and self-supervised learning have similarities in that they both aim to learn reasonable representations from unlabeled data. The main difference between them is whether they leverage labeled information in the process of representation learning or not. In this work, we also validated self-supervised learning methods, such as SelfHAR [45], which showed lower performances compared to the semi-supervised methods. This might be due to the risk of trivial representations learned by self-supervised learning. For example, the constraint of learning representation in SelfHAR is the disagreement among different augmented views; however, the pre-task is not challenging enough as the model can achieve very high accuracy in distinguishing augmentations and might have learned trivial representations. Thus, we speculate that semi-supervised learning constrains the learning process more effectively with labeled samples in the training set.
In this study, we conducted empirical experiments on SOTA semi-supervised learning approaches in stress detection and human activity recognition tasks. Further, by proposing a semi-supervised learning framework with a novel hinge of active sampling method, we achieved even higher performance than the reproduced methods in the literature. We believe that our work can inspire future studies in wild stress or other human construct detection. Since label annotation is usually expensive, the lack of training samples becomes a common problem in this field. To the best of our knowledge, there are not many semi-supervised learning studies conducted on stress detection. Our work, on the other hand, significantly improves the performance of the model on top of the baseline. Therefore, we believe that this study can help other future work to achieve more accurate stress detection results and thus help more people to solve stress-related problems.
Moreover, this work can contribute to other sensor data-based recognition tasks in the IMWUT community. Ubiquitous computing is a promising topic that involves sensors that passively measure data. Many of the applications can meet the same challenge in the limited number of labels compared to passive sensor samples. Thus, we believe that our work has the potential to be extended to multiple applications in this field.
In this work, we have implemented several algorithms including the baselines, reproduced SOTAs, and the proposed semi-supervised methods. Although we tried to tune the model structure and hyperparameters for the evaluation datasets accordingly, however, there might still be room for improvement regarding the model performances. In addition, interpretability and explainability can be important when considering deploying the model into clinical applications. We have not explored interoperability at the current stage, which can be considered another limitation and future work.
In this work, we examined the SOTA semi-supervised algorithms and proposed a semi-supervised learning framework to help human stress estimation by leveraging massive unlabeled physiological and behavioral data collected in the wild. Our proposed method contained components of active sampling, auto-encoder pre-training, and consistency regularization. We evaluated our proposed methods using two datasets with a small amount of labeled data but a massive amount of unlabeled samples. We demonstrated that our proposed active sampling approach helped boost the performance of stress detection with fewer unlabeled samples. Our results showed that fully leveraging our proposed methods provided the best results in accuracy rate and F1 scores on both datasets. In addition, our ablation experiments on four human activity recognition datasets demonstrated that the proposed method improved the model performance compared to the supervised learning approach substantially. In the future, we will continue developing our method to improve its performance on datasets with categorical features and improve the interpretability and explainability of the methods.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:3
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:5
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:9
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:11
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:15
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:17
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:19
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 7, No. 2, Article 80. Publication date: June 2023. Semi-Supervised Learning for Wearable-based Momentary Stress Detection in the Wild • 80:21