Ubiquitous computing is entering almost every aspect of our life. As close companions to users, smartphones and wearable devices can continuously and passively capture various aspects of daily behavior. In the past decade, a large body of work has demonstrated the capability of longitudinal passive sensing and behavior modeling in many application areas, such as detecting physical health issues [5,60,65,104], monitoring mental health status [96,101], measuring job performance [57,61], tracking education outcomes [97,114], and tracing social justice [82]. Researchers have used a variety of approaches to address their research questions, ranging from doing statistical analysis to building machine learning (ML) models.
Due to the high cost of conducting longitudinal passive sensing studies, most prior research collected selfreported surveys data (as ground truth) and sensor data from one population over a few weeks or months. Their data analysis and models mainly focused on a single dataset. However, to build a model with practical and useful deployability, it is essential for researchers to evaluate the model across multiple datasets to ensure its generalizability. As real-life deployment will introduce new populations under new contexts, the model needs to generalize well to unseen data with robust performance. The gap between the single dataset-based analysis and the need for cross-dataset evaluation has been long-standing in the longitudinal behavior modeling domain. Moreover, the generalization challenge could come from multiple sources, such as different populations, different users in the same population, and the same users across different time periods. It remains unknown to the community which of these issues makes cross-dataset generalization particularly challenging. In this work, we take the first step towards a systematic evaluation of behavior models' generalizability 1 across multiple datasets, using depression detection as an example. We call for researchers' attention to incorporate this critical step into model analysis pipeline in the future.
To perform an analysis of model generalizability, we formed a collaboration of two research groups across two institutes. Each group conducted two rounds of longitudinal passive sensing studies (one in 2018 and one in 2019, before the impact of the COVID-19 pandemic), focusing on depressive symptom detection. At each institute, there was a small fraction of overlapping users across the two years. We employed a uniform data transformation and feature extraction process to build four institute-year datasets, and re-implemented a large proportion of behavior features in prior depression detection studies (e.g., [19,80,93,98,100,101]). We then evaluated the cross-dataset generalizability of behavior models using the four datasets. In this paper, we focused on a common binary classification task as a starting point: distinguishing whether participants had at least mild depression.
We closely re-implemented 9 prior depression detection algorithms using overlapping features from the four datasets, and tested the models generated via these algorithms, using a consistent cross-validation setup on each dataset. Although these models outperform the naive baseline predicting the majority (Δ = 7.0 ± 7.6%%), we observed a substantial drop between these models' performance on our datasets and their reported results in literature (Δ = 18.9 ± 9.7%), which indicates the lack of cross-dataset generalizability of these algorithms.
Second, a range of domain generalization algorithms were developed with recent advances in the ML community [94,118], which were mainly designed for computer vision (CV) or natural language processing (NLP) tasks. However, there is no prior work applying these methods to the longitudinal behavior modeling domain. To test these methods, we re-implemented 8 types of recent deep-learning-based domain generalization algorithms, and built 15 different models using these algorithms (an algorithm can generate multiple models). We used a leave-one-dataset-out setup to evaluate the cross-dataset generalizability of prior depression detection methods and recent domain generalization algorithms, i.e., using one dataset as the testing set and the rest of datasets as the training set. However, our experimental results revealed that these models did not generalize well to our longitudinal passive sensing domain. The best model only has a minimal advantage on ROC AUC (54.1%) over the naive majority baseline (50.0%, Δ = 4.1%). It is worth noting that our main goal is not to criticize or single out any prior work (including our own), but to emphasize the importance and lack of model generalizability for longitudinal passive sensing of human behaviors.
We further present two new algorithms to enhance model generalizability. The first method, Clustering, extends a Siamese network and builds an ensemble model based on unsupervised clustering, which could utilize the similarity of behavior trajectories among individuals. The second method, Reorder, creates a new task of solving a temporal reordering puzzle in addition to the main depression detection task, which was inspired by selfsupervised techniques in CV and NLP approaches [24,77]. Reorder is forced to learn the continuity of behavior trajectories and achieve better generalization. Our results demonstrate that Reorder significantly outperforms other methods by at least 3.4% on ROC AUC (6.3% relative advantage) and 3.2% on balanced accuracy (6.2% relative advantage) in the leave-one-dataset-out evaluation. Moreover, our datasets support multiple cross-dataset tasks besides leave-one-dataset-out, including different populations (two institutes), different users within the same population (same institute across two years), and same users at different times (overlapping users across two years). Reorder consistently achieves the best or the second best performance under other cross-institute and cross-year generalization tasks. Meanwhile, Clustering achieves the highest upper bound when the model training can early stop at the optimal epoch [112]. These results indicate the strong generalizability advantage of our proposed techniques. Nevertheless, we acknowledge that the advantage of our new methods is statistically significant yet incremental. More future work will be needed to further advance the area.
In addition, the comparison between multiple generalization tasks enables us to investigate the major challenge of domain generalization in our dataset. We identify that compared to the time period difference (for the same user), individual difference (i.e., user-user difference, either from the same or different populations) is the main source of various generalization challenges.
We consolidated our implementation of 19 methods (9 prior depression detection algorithms + 8 recent domain generalization algorithms + 2 new algorithms) and developed a benchmark platform, GLOBEM (short for Generalization of LOngitudinal BEhavior Modeling), to accelerate cross-dataset evaluation by others in future research. Compared to existing cross-dataset generalization platforms (DomainBed [36], DeepDG [94]) and benchmarks (WILDS [48]), our platform further extends the domain generalization problem to the longitudinal passive sensing field. GLOBEM not only supports flexible and rapid evaluation of the existing methods, but also provides easy-to-extend templates for researchers to develop and prototype their own algorithms and systems.
The contributions of our work can be summarized as follows:
• We highlight the importance of cross-dataset generalizability evaluation of longitudinal behavior models. Joining forces across two research groups, we collected four passive sensing datasets with similar sensors and labels, using depression detection as an example. To the best of our knowledge, we are the first to conduct a comprehensive multi-dataset evaluation of existing behavior modeling algorithms. • We re-implemented 9 prior depression detection algorithms and 8 modern domain generalization algorithms.
Our experiments revealed the general lack of generalizability of these methods in our data format. We envision our work can inspire researchers in the ubiquitous computing and ML communities to put continued and additional effort into building behavior modeling algorithms that are robust and generalizable across multiple domains and datasets.
In this section, we first briefly summarize prior work on depression detection based on longitudinal passive sensing (Section 2.1). We then provide a high-level overview of the recent advances in the ML community on domain generalization (Section 2.2).
Depression, formally known as major depressive disorder (MDD), is a common and important mental health problem that affects people worldwide. Research has found that depression affects approximately 216 million people globally [92]. A recent report estimated that 7.2% of all U.S. adults had at least one depressive episode in a year, i.e., a period of more than two weeks of depressive symptoms [72], and the prevalence has increased during the pandemic [44,66]. Early detection of depression can potentially assist in mitigating or preventing its detrimental consequences. There is a growing realization that everyday devices can help us understand the relationship between people's daily actions and depressive symptoms, by continuously and passively collecting daily behavioral data [4,10,14]. The success in the last decade of using mobile sensing for depression research has attracted increasing attention of researchers from various communities [49,103].
Earlier research focused on understanding the statistical relationship, such as correlation, between depressive symptoms and mobile sensing data [11,80]. For example, Saeb et al. [80] identified a significant correlation between depression scores and features related to location and phone usage. Ben-Zeev et al. [11] found a significant correlation between changes in depression severity levels and features related to sleep duration, speech duration, and mobility. Recently, researchers have leveraged the results of correlation analysis to build ML models for depression diagnosis and detection [19,69,93]. For instance, Farhan et al. [27] employed location features to detect biweekly depression and their top model achieved an F1 score of 82.0% on an eight-month dataset with 79 college students. Wang et al. [98] hand-crafted several cross-sensor features using data from mobile and wearable devices. On a dataset collected from 68 college students across two nine-week terms, their best model achieved 81.5% recall and 69.1% precision. Xu et al. [100] proposed an automated feature extraction pipeline across multiple sensors' data. They used association rule mining to discover frequent behavior rules and extract interpretable features. Their model achieved an accuracy of 81.8% and an F1 score of 84.3% on a dataset with 138 college students over 16 weeks. They further developed a personalized depression detection model using a new method based on collaborative-filtering [101]. Their model achieved an accuracy of 82.4% and an F1 score of 85.5% on a similar dataset.
However, most existing depression detection research have mainly developed and evaluated their models on only a single dataset. There is a lack of understanding and evaluation of these models' generalizability across multiple datasets. We aim to address this gap in this paper.
Building a model that can generalize across multiple domains or datasets has been a challenging problem in the ML community. This is one of the problems that transfer learning aims to address, namely domain adaption, which focuses on cases where the model can access some data of the target domains (e.g., datasets) in addition to the data of the source domains [74,95]. A more challenging task is domain generalization, where the model can only access source domains [118]. In this paper, we focus on the domain generalization task with datasets that have homogeneous feature spaces and label spaces but with divergence [115]. Such a task corresponds to an ideal real-life deployment setup: directly applying a trained model to new users, without the need to access any data from the new users.
In the past few years, ML researchers have proposed a wide range of algorithms for domain generalization. Most of them belong to one of the following three categories [94]: 1) Data manipulation, which enhances the data by augmentation or generation techniques to assist the model training (e.g., [23,113]); 2) Representation learning, which aims to learn desirable feature representations that can generalize across domains (e.g., [2,28,31]); 3) Learning strategy, which focuses on exploiting the training procedure to promote a model's generalizability (e.g., [50,88,111]). We refer readers to recent domain generalization surveys for more details [94,118]. Almost any applied ML area will encounter the cross-dataset domain generalization challenge for real-life deployment, such as object recognition [6,15,45,86], NLP [70,117], affective computing [20,42], security [41,116], and intelligent interaction [102,105,106]. Researchers have developed cross-dataset benchmark platforms such as DomainBed [36], DeepDG [94], and WILDS [48] to facilitate related studies in the ML community. Recently, researchers started to merge multiple passive sensing datasets for behavior model training [1]. However, there is no prior work on cross-dataset generalizability evaluation in the longitudinal passive mobile sensing area. Perhaps the most relevant work is by Mishra et al. [62]. They evaluated three models' generalizability in the physiological stress detection domain using four datasets collected from different wearable devices in a passive manner. However, their models aimed at short-term detection (over a few seconds), so the ground truth labels were frequent and rich. Moreover, the datasets they employed were all collected in a lab setting. Instead, we focused on in-the-wild longitudinal passive sensing problems, where the ground truth is usually much more sparse (e.g., once per one to two weeks or less frequent for depression labels [14,100]), and the field experiments were not controlled. To the best of our knowledge, we are the first to conduct an extensive cross-dataset evaluation analysis in our area.
We combined passive sensing data from two institutes collected by two research groups, each across two years (before the impact of the pandemic). We first introduce the data collection process (Section 3.1), and then describe the feature extraction steps that were employed consistently across all of the datasets (Section 3.2).
Both research groups employed the data collection model proposed in [96]. The datasets were collected from two different Carnegie-classified R-1 universities in the United States using a similar method [43,82]. In both institutes, the data collections went through an IRB review and approval. We recruited undergraduate students via emails and social media posts. We collected four datasets in total, two from each institute (one in 2018, and one in 2019). Every dataset contained data from participants over the same academic quarter for 10 weeks (Spring quarter, approximately from late-March to mid-June) 2 , so the impact of seasonal effects was controlled. We summarize the study details in Table 1.
The four datasets have 155, 218, 93, and 152 students (618 person-years in total), respectively, with a high percentage of female students. There were a few overlapping participants who were in both years' datasets (23 in Institute 1, and 58 in Institute 2); thus the datasets contain 534 unique participants in total. Every quarter, Table 1. Basic Study Information and Participant Demographics of Four Datasets. Participants with less than 2 weekly EMAs or less than a quarter of their sensor data (i.e., missing rate > 75%) were excluded from the dataset. In the ground truth row, the percentage in the parentheses indicates the proportion of participants having at least mild depressive symptoms based on the corresponding questionnaires. Gender acronym -F: Female, M: Male, NB: Non-binary. Racial acronym -A: Asian, B: Black or African American, H: Hispanic, N: American Indian/Alaska Native, PI: Pacific Islander, W: White, NA: Did not report. & is used when participants reported more than one races. participants received up to $245 (Institute 1) and $100 (Institute 2) for compensation, based on their compliance. The studies at Institute 1 also used a wearable Fitbit for data collection, and participants were allowed to keep the Fitbit after the study if their study compliance was high enough.
3.1.1 Ground Truth. In both institutes, we employed well-established and validated questionnaires to obtain ground truth of participants' depressive status. We collected survey results on a weekly basis, and an end-term survey at the end of the quarter. These surveys were delivered via smartphones. The weekly surveys include PANAS [22] (Institute1Year1 only), PHQ-4 [54] (remaining datasets). The end-term surveys include BDI-II [9] (Institute1) and PHQ-4 (Institute2). PHQ-4 and BDI-II are both screening tools for further inquiry of clinical depression diagnosis. They distinguish four severity levels: no or least, mild, moderate, and severe depressive symptoms. As an initial step of model generalizability evaluation, we focus on a binary classification task to distinguish whether participants' scores indicate at least mild depressive symptoms through the scales (i.e., PHQ-4 > 2, BDI-II > 13) 3 .
Note that although PANAS contains questions related to depressive symptoms (e.g., "distressed"), it does not have a comparable theoretical foundation for depression detection like PHQ-4 or BDI-II. Therefore, to maximize the compatibility of the datasets, we trained a small decision tree (depth=2) on Institute1Year2 data that has both PANAS and PHQ-4 scores to generate reliable ground truth labels. We used PNANS scores (on a 1-5 Likert scale) on two affect questions, depressed and nervous, as the input and PHQ-4 score-based depression binary label as the output. Our model achieved an accuracy of 74.5% and an F1-score of 76.3% on a 5-fold cross-validation. It generated a simple rule: the user would be labeled as having at least mild depression when the distress score is greater than 1, or the nervous score is greater than 2. We then applied this rule to Institute1Year1 dataset to generate labels. We discuss potential limitations of our label generation method in Section 6. After this procedure, all datasets contain binary depression labels for both end-term and repeated weekly prediction tasks. The average number of total depression labels is 12.9 ± 4.1 per person.
3.1.2 Passive Sensing Data. The two research groups employed two separate smartphone applications for passive sensing data collection. In Institute 1, the app was developed based on the AWARE Framework [29]. In Institute 2, the app was developed based on the work by Wang et al. [96]. Both apps are compatible with iOS and Android systems. The functionalities of these two apps are very similar: they both could collect location, phone usage (screen status), physical activities, and sleep patterns. The app of Institute 1 also recorded a phone's nearby Bluetooth and WiFi addresses, as well as call logs. The app of Institute 2 further recorded audio data via microphone. Moreover, the studies in Institute 1 employed a wearable Fitbit to collect participants' detailed activities and sleep behaviors, while the studies in Institute 2 combined multiple sensors (e.g., accelerometer, gyroscope, microphone, and lightness sensor) to infer activity and sleep behaviors [18,96].
To ensure the consistency and reproducibility of features extracted from the multiple datasets, we utilized the RAPIDS [91] platform that supports feature extraction from multiple passive mobile sensors with flexible time windows. RAPIDS is open source to enable Reproducible Analysis Pipeline for Data Streams. It supports data streams logged by multiple platforms (smartphones, Fitbit wearables, and Empatica wearables), integrates multiple previous research efforts (e.g., [8,14,25]), and has been leveraged in a range of passive sensing and behavior modeling studies (e.g., [90,109]). As RAPIDS builds a feature extraction pipeline based on data collected by the AWARE framework, we transformed Institute 2's data format to make it compatible with RAPIDS, without altering the raw data content. We used the same RAPIDS feature extraction configuration across all datasets.
One of our goals is to reproduce a list of previous depression detection models. Therefore, our feature extraction focuses on re-implementing features that occur in previous work. RAPIDS already provides a wide range of features that we can easily leverage 4 . We extended it to cover features of the four most important modalities across the four datasets: location, phone usage, physical activity, and sleep behaviors. We excluded features that were not common in all datasets to ensure maximum compatibility, including features related to Bluetooth, WiFi and calls in DS1 and DS2, as well as audio features in DS3 and DS4.
3.2.1 Data Type: Location. We included the all location features provided by RAPIDS, including travel distance, location variance, location entropy, moving speed, number of frequent locations visited, radius of gyration, etc. We also incorporated more location features (duration of staying and percentage of total time) related to specific points of interest, including places for living/home, study, exercise, and relaxation.
Type: Phone Usage. In addition to screen-related features in RAPIDS about the statistics of unlocking episodes (count, sum, mean, standard deviation, maximum, minimum), we further considered these features at different locations (home and study places) to contextualize the phone usage behavior.
Type: Physical Activity. We leveraged the Fitbit-based activity features in RAPIDS, which include 1) high-level summary features like the number of steps, duration of being active, 2) low-level features that consider the statistics (mean, standard deviation, maximum, minimum) of active or sedentary episodes.
Data Type: Sleep. We utilized the Fitbit-based sleep features in RAPIDS. Similar to activity features, sleep features also include 1) high-level summary features such as total duration of being asleep or in bed, 2) low-level features that calculate the statistics (count, mean, maximum, minimum) of episodes of being asleep, restless, and awake during the night.
. We incorporated all of them when extracting the features in Section 3.2.1-3.2.4, including four epochs of a day (morning 6 am -12 pm, afternoon 12 pm -6 pm, evening 6 pm -12 am, and night 12 am -6 am) [19,82,101], the whole day [19,80,93,98], and past two weeks [14,27,55,93].
Post-processing. After a consistent feature extraction process, we prepared four datasets with the same feature and label spaces. Some prior work also involved the procedure of feature normalization, so we doubled the feature number by adding normalized features based on each individual's distribution: subtracting the median and scaling with the 5-95 quantile range on each individual. The current data format for each participant is a time-series feature-vector data (i.e., a long feature matrix), accompanied by a short list of labels on certain dates. Since participants may have a different number of labels, we further sliced the feature sequence based on labels to make consistent data input shapes. Specifically, given a label collected on a particular day, we sliced a feature matrix of the past four weeks, counting back from the day (28 days × feature number). We picked four weeks in order to cover previous depression detection models' feature calculations [14,27,55]. After the slicing, every participant label corresponded to an input feature matrix with the same shape. Note that missing features (due to the missing raw data, accounting for 7.3±2.3% of the data) were imputed with the median value. Once we finished post-processing the feature data, we could then evaluate various algorithms across these datasets 5 .
We developed a uniform benchmark platform and compared 19 methods' generalizability on our four datasets. We first introduce prior behavior models that focused on depression detection in Section 4.1. We then describe the eight recent deep learning-based domain generalization algorithms in Section 4.2. We highlight our newly proposed methods that demonstrated improved generalizability in Section 4.3. All of these methods were evaluated on a uniform platform that we developed and plan to open-source (Section 4.4).
We introduce the nine prior depression detection algorithms with technical details. We re-implemented the algorithms with the same or similar features, and evaluated these models on our datasets. 6 The specific model setup can be found in the configuration files in our codebase.
(1) Canzian et al. [14]: used location trajectory features directly computed from the past two-week time window to train a support vector machine (SVM) for depression detection. (2) Saeb et al. [80]: used the combination of location and screen features and aggregated their daily average of the past two weeks to train a logistic regression model with elastic regularization. (3) Farhan et al. [27]: used location and physical activity features from the past two-week window to train an SVM model. (4) Wahle et al. [93]: used features from several sensors (activity, location, WiFi, screen, and call) over the past two weeks. They used both daily aggregation (i.e., mean, sum, variance) and direct computation of the features of the two weeks to build SVM and Random Forest models. We left out the WiFi and call features to ensure its compatibility with our datasets. (5) Lu et al. [55]: used location, activity, and sleep features computed from the past two weeks and built multi-task learning models combining linear regression and logistic regression. To further deal with device platform differences, they built one model for iOS devices and one for Android devices. (6) Wang et al. [98]: used location, screen, activity, sleep, and audio features and aggregated their daily average and slope of the past two weeks (for the frequent prediction) or the whole study period (for the end-of-term prediction). They built a lasso-regularized logistic regression model for the prediction. We excluded audio features as they were not collected in all datasets. (7) Xu et al.-Interpretable [100]: used location, screen, activity, and sleep features in multiple epochs of a day (morning, afternoon, evening, night). They first applied association rule mining to mine out interpretable behavior rules that capture differences between participants with depression and without depression. Then, they used the rules to filter and aggregate features of multiple days and built an Adaboost model for the detection. ( 8 -Personalized was developed on DS1 and DS2, using a more strict data filtering criteria (resulting in a smaller user group). However, the original work employed additional feature types (Bluetooth, WiFi, call, message, and battery) and these features were not included in this work, to allow for comparisons across all of the datasets.
Previous depression detection algorithms did not focus on the domain generalization challenge. In the ML community [94,118], a range of domain generalization algorithms were proposed recently, but they were mostly designed for CV/NLP tasks. There is no prior work evaluating these methods on longitudinal passive sensing datasets. Therefore, we implemented eight well-studied deep learning techniques to cover the major approaches of domain generalization [94], including 1) data manipulation (Mixup [113]), 2) representation learning (IRM [2], DANN [31], CSD [76]), and 3) learning strategy (MLDG [50], MASF [26], Siamese [47]). The input-output format of these models is the same: we picked a subset of important daily features in the most recent traditional depression detection algorithms [19,101]. We then used the past-four-week feature matrix as the input, and the depression label as the output.
(1) ERM (Empirical Risk Minimization) [89]: the basic model training techniques without particular design for domain generalization. ERM shows a competitive performance in previous CV generalization tasks [36,94]. We implemented multiple architectures with ERM: a) ERM-1D-CNN: one-dimensional CNN that treats the data as a time series of length 28; b) ERM-2D-CNN: two-dimensional CNN that treats the data axs an one-channel image; c) ERM-LSTM: another architecture to model time-series data; d) ERM-Transformer: a transformer-based architecture for modeling sequence data. (2) Mixup (ERM-Mixup) [113]: a popular data manipulation and augmentation technique that performs linear interpolation between any two instances with a weight sampled from a Beta distribution. Mixup can be plugged into any model architecture and training pipeline. In this paper, we used 1D-CNN in our experiment as it has a similar performance as 2D-CNN (as shown in Section 5), while more robust to feature positions in the feature matrix. Similarly, we also used 1D-CNN for the other methods in the rest of this section if they are agnostic of architectures. (3) IRM (Invariant Risk Minimization) [2]: a representation learning paradigm to estimate invariant correlations across multiple distributions and learn a data representation such that the optimal classifier can match all training distributions. ( 4) DANN (Domain-Adversarial Neural Network) [31]: another representation learning technique that adversarially trains the generator and discriminator. The discriminator is trained to distinguish different domains, while the generator is trained to fool the discriminator to learn domain-invariant feature representations.
For our purposes, we treated each dataset as a domain (DANN -Dataset as Domain), or each person as a domain (DANN -Person as Domain). 7) MASF (Model-Agnostic Learning of Semantic Features) [26]: a learning strategy that combines metalearning and feature disentanglement. After simulating domain shift by domain split, MASF further regularizes the semantic structure of the feature space by introducing a global loss (to preserve relationships between classes) and a local loss (to promote domain-independent class clustering). We tested MASF -Dataset as Domain, and MASF -Person as Domain. (8) Siamese Network [47]: a metric-learning based strategy to find a better pair-wise distance metric. It aims to decrease the distance between positive pairs (i.e., same labels) and increase the distance between negative pairs (i.e., different labels).
We observe gaps in the existing algorithms. First, prior depression detection algorithms (Section 4.1) did not attempt to address the domain generalization challenge. In addition, recent domain generalization models proposed by the ML community (Section 4.2) were not specifically designed for longitudinal passive sensing data. Therefore, we proposed two new methods that leverage human behavior characteristics from two different perspectives: one using the similarity of behavior trajectories among different individuals (Section 4.3.1), and the other leveraging the continuity of behavior trajectories of each individual (Section 4.3.2).
Trajectories to Train Independent Models. This method stems from a simple intuition: Although participants came from different populations, certain periods of some users' behavior trajectories could be similar and clustered together. Data in the same cluster might have closer distributions and simplify the classification task within the cluster. Specifically, we built an ensemble model based on unsupervised clustering. Using all training data, we first followed [38,99] to train a deep clustering model with convolutional auto-encoders (DCEC) that assigns data points with cluster indices. For each cluster, we then used data in that cluster to train a small Siamese model [47]. In the inference stage, data would be fed into the DCEC model to find the cluster index, and classified by the corresponding Siamese model.
The Model to Learn Behavior Continuity. The challenge of domain generalization is mainly caused by the data distribution shift in heterogeneous domains. In our case, such a shift comes not only from dataset differences (i.e., each subpopulation behavior pattern varies [21,59]), but also from individual differences (i.e., each person behaves uniquely [67,83]). However, despite these differences, there still exists a range of similarities among individuals' behaviors. For example, people tend to have daily routines, which define the structure of and influence of almost every aspect of everyday behaviors [7]. Although individuals have unique routines, these patterns would lead to continuous or even repetitive behavior trajectories from day to day [40]. Such an observation motivates us to leverage the behavior continuity and construct a self-supervised learning task to obtain generalizable feature representations.
Self-supervised learning is a recently popular learning paradigm that builds self-supervised tasks (i.e., pretext tasks) from the unlabeled data. The pretext tasks are often not directly related to the main prediction task. It has been applied to domain generalization problems in CV tasks (e.g., JiGen [16], SelfReg [46]) and NLP tasks (e.g., BERT [24]). For example, BERT used a pretext task where words in a text are randomly masked and the goal is to predict these words.
To leverage the continuity of behavior trajectory, inspired by [16], we proposed a new multi-task learning model, Reorder, with a new pretext task called reordering puzzle (see Figure 2): we shuffled the temporal order of the feature matrix, and trained a model to reconstruct the original sequence, jointly optimized with the main classification task over different domains. The model needs to achieve two tasks simultaneously: 1) it will learn to capture the continuity of behavior trajectories, so that it can find the right temporal order of the time-series feature data after shuffling; and 2) it will also learn to solve the main task (i.e., depression detection in our case). Due to the prevalence of the continuous behavior trajectories based on human nature (analogous to the continuous edges and patterns in images [16]), solving the first task by learning such continuity could assist the model to extract more generalizable representations of behavior trajectories across individuals. This could potentially enable more robust domain generalization in behavior modeling.
Specifically, we created a multi-task learning model function ℎ, with the 1D-CNN based embedding (parameters 𝜃 𝑓 ), fully connected layers for reordering (parameters 𝜃 𝑟 ), and fully connected layers for classification (parameters 𝜃 𝑐 ). The first task is the main classification task. The loss function of this task is L 𝑐 (ℎ(𝑥 |𝜃 𝑓 , 𝜃 𝑐 ), 𝑦), where 𝑥 is the input matrix, and 𝑦 is the classification label. The second task is the reordering task. We first sliced the feature matrix along the temporal dimension into 𝑛 segments and then shuffled these segments. We picked the number of segments 𝑛 = 10 ( ⌈28/3⌉) since 28! or 14! (28/2) is too computationally expensive. Moreover, as 10! total possible permutations is an overly large number, we followed the practice in [16] and predetermined a subset of 𝑃 = 200 permutations by following the Hamming-distance-based method [71]. We then assigned an index to each permutation. Within the subset, the reordering task is equivalent to identifying the index of the permutation, which is essentially a classification task. Therefore, the loss function of the reordering task is L 𝑟 (ℎ(𝑧|𝜃 𝑓 , 𝜃 𝑟 ), 𝑝), where 𝑧 is the feature matrix 𝑥 after the reordering, and 𝑝 is the permutation index. Overall, the model can be [113] to increase the variation of the data. It is worth noting that the reorder puzzle is only enabled during the training stage. There is no shuffling at the testing stage to avoid extra noise.
To gain a fair and reliable performance of these algorithms, we built a benchmark platform, GLOBEM, to incorporate all algorithms mentioned from Section 4.1 to 4.3. Compared to the existing platforms DomainBed [36] and DeepDG [94] that mainly aim for image-based domain generalization tasks, GLOBEM specifically focuses on longitudinal passive sensing data.
Figure 3 illustrates the overall structure of the GLOBEM platform. It splits the whole pipeline into three independent modules:
(1) The feature preparation module defines behavior features used by the algorithm;
(2) The model computation module defines how a behavior model is going to be trained. These two modules are determined by the core algorithm; (3) The configuration module provides the flexibility to adjust hyperparameters in the algorithm. Researchers and developers can re-use or re-purpose any of these modules to develop new algorithms within the pipeline. Moreover, GLOBEM separates the configuration setup from the model definition, supporting easy testing and ablation studies of hyperparameters and different features. We summarize our analysis results in this section. We first reveal the feature difference among datasets (Section 5.1). We then demonstrate the significant performance drop between the existing depression detection models' performance on our datasets versus their reported results in previous literature (Section 5.2). We conducted cross-dataset evaluation to compare the generalizability of all methods, and demonstrated the advantage of our proposed Reorder (Section 5.3-5.6). Finally, we summarize our key findings in Section 5.7.
We analyzed the commonalities and differences between the features in the four datasets (Section 5.1.1) Then, we conducted two tasks to quantify the distribution difference among the 1) datasets (Section 5.1.2) and 2) individuals (Section 5.1.3).
5.1.1 Feature Analysis. We started with the analysis between every single feature's value and the prediction target, i.e., depression labels. We computed one linear mixed effect model for each feature (with participant ID as the random effect) within each dataset, and identified those significant features (𝑝 < 0.05) having the same directions in all datasets (i.e., either all positive or all negative coefficient). Figure 4a summarizes representative features with large coefficients in each data type. For example, the sleep-related feature coefficients show that lower sleep duration and more interrupted sleep (as indicated by the count of asleep episodes) are associated with higher depression scores. We will discuss these results more in Section 6.
Figure 4b presents the distribution of some representative features of different data types. Some features have similar patterns. For example, the duration of being asleep has a peak around 7 hours, but the variance is different across datasets. Some features have similar patterns within the same institute. For example, DS1 and DS2 have closer distributions on the number of steps, while DS3 and DS4 have closer distributions, which could be related to location contexts of each institute. Meanwhile, some features have distinctive patterns. For example, the average number of frequent locations visited in DS1, DS3, and DS4 have two peaks, while only one peak is visible in DS2.
These findings indicate that the same feature across datasets can often have various distributions.
To quantify the difference among datasets, we first conducted a "Name-The-Dataset" task on our four datasets [86]. For every dataset, we first aggregated each feature matrix by calculating the mean along the time dimension to obtain a feature vector for each sample. We then performed an 80%/20% user split for the training/testing set, i.e., no overlapping user's data is in both the training and testing sets simultaneously. We used a portion of the training data to train a small random forest model (10 decision trees, each with a maximum depth of 3) to classify which dataset a data belongs to (i.e., four-class classification).
We used SMOTE to mitigate the data imbalance as datasets have different sizes [17].
The left side of Figure 5a indicates the model performance on the testing set. With only 1 user in the training set (around 0.2% of samples from the training set), the model is able to achieve an average accuracy of 62.0%, compared to 25% as the baseline. With 5 users (1%) and 50 users (10%) from the training data, the accuracy reaches 81.7% and 96.8%, respectively. These results indicate that the behavior features from different datasets (i.e., populations or years) have different distributions allowing them to be easily differentiated.
Feature normalization is one of the common techniques for mitigating feature value differences and aligning the data. Therefore, we also trained another model with normalized features (subtracting the median and divided by the 5-95 quantile range). As shown in the right side of Figure 5a, the model still achieves an accuracy of 34.0%, 49.9%. and 67.8% with 1, 5, and 50 users from the training set. The normalization does diminish the distribution shift, but the distinguishable differences between datasets still persists.
To quantify the differences among individuals, we further conducted a "Distinguish-The-Person" task, replacing the label from the dataset with the user ID. We performed an 80%/20% split on each user's data for the training/testing set. Similar to the "Name-The-Dataset" task, we then used a proportion of the training data to train another random forest model (maximum leaves=2k) to classify which user ID a data point belongs to, which is a challenging 534-class classification task (same as the total number of unique participants in the four datasets).
The two plots in Figure 5b show the performance using the features before and after normalization. Using 1, 5 and 10 data points per user from the training set, the model achieves 24.7%, 74.8%, 87.4% using direct features, and 12.2%, 33.5%, 50.5% using normalized features, which are all significantly higher than the baseline accuracy 0.19% (1/534). We also tested the person-year classification (618 user-years), which showed similar results. These analysis results clearly show that data from different datasets and individuals all have distinguishing distributions. In the "Distinguish-The-Person" task, the relative advantage over baseline is even larger than the "Name-The-Dataset" task, suggesting that the challenges of domain generalization in longitudinal human behavior modeling may come more from individual differences than dataset or population differences. We will present more results to support this claim in Section 5.4.
We followed the same procedure as the prior depression detection work to evaluate the performance of these 9 models on each of the four datasets. Some models focused on end-of-term detection [19,27,80,100,101], while others focused on frequent weekly detection [14,55,93,95]. Thus we evaluated these models on both tasks. To keep the process consistent with the prior work, the models' hyperparameters were tuned via grid search with the same range as mentioned in each prior work. The training and testing used data from the same dataset, using twenty-fold cross-validation at the user level (i.e., leaving 5% of the users out in each fold). For each of the four datasets, we repeated this process and calculated balanced accuracy as the metric, i.e., the mean of sensitivity (true positive rate) and specificity (true negative rate). We picked this metrics for two reasons. First, most of the prior work reported metrics based on the confusion matrix (e.g., precision, reall, F1 score), thus we can easily calculate and compare balanced accuracy from the prior work. Second, it has been shown more robust to class-imbalance compared to accuracy or F1 score [12].
Table 2 summarizes the results of model performance on each dataset. For the end-of-term depression detection, only three models, Xu et al.-Interpretable, Xu et al.-Personalized, and Chikersal et al., achieve a balanced accuracy over 60%. However, these models' performance is significantly lower than the reported result in the prior work (average Δ = 15.9 ± 10.7%). We have similar findings in the repeated weekly depression detection task. None of these models' performance reaches 60%. Again, we observe a big gap between the reported results and our results (average Δ = 22.6 ± 8.5%). Figure 6 highlights the performance drop of these methods. Such findings indicate that prior algorithms do not generalize well on our datasets.
Table 2. Balanced Accuracy of Predicting End-of-term or Weekly Depression Status. Models are evaluated by 5-fold crossvalidation within each of the four datasets. The "Prior Work" columns show the performance contrast between the reported results in prior literature (evaluated on their own datasets) and the results on our datasets. + / ++ means the literature only reported F1-score/ROC AUC, which was usually close to balanced accuracy.
Task1: End-of- Some features may be particularly tuned to specific datasets, which leads to feature overfitting to a certain group of participants in one dataset. These features cannot properly generalize for another group of participants from a different population and dataset. Thus, we conduct cross-dataset evaluation experiments to systematically measure the models' generalizability.
One of our core research goals is to explore the generalizability of behavior models. Using four datasets, we built a leave-one-dataset-out evaluation pipeline [73,78]. Specifically, each time we took out one dataset as the testing set, and used the three other datasets as the training set. Such a cross-dataset analysis can effectively measure how a model trained on existing datasets could work on a new unseen dataset [50,86].
We focused on the weekly depression prediction as the main task, because the small sample size of the end-of-term task makes it infeasible to train deep models, i.e., all of the models (17) other than the prior depression detection models (9). For the prior depression detection models, we followed a similar procedure as Section 5.2 to conduct a hyperparameter tuning by grid search on the three training datasets.
The rest of the methods (Section 4.2 and 4.3) all used deep models. For methods that used 1D-CNN as the backbone, we used a simple architecture based on a small-range tuning using ERM-1D-CNN: It had three 1Dconvolution layers (size 8, stride 3, ReLU activation), each followed by a batch normalization layer, a max-pooling layer, as well as a dropout layer (rate 0.25). A fully connected layer (size 16) was attached after flattening the third convolution layer's output to convert it into a vector of length 16. The following layers were then customized for each model.
For our new method Reorder, the feature layer was connected to two fully connected layers (size 16 and 2) for the classification task, and another two layers (size 32 and 200) for the reordering task. Additional model details are listed in codebase's config files. For the new method Clustering, we picked the cluster number as 60. The auto-encoder had two 1D-convolution layers (size 64 and 32) with the middle hidden size as 10. The Siamese network in each cluster adopted the same three 1D-convolution layers (size 8) architecture as other models.
Other architectures were also kept simple: 2D-CNN used three 2D-convolution layers with the same size, stride, and activation function as 1D-CNN; LSTM used two bi-directional layers (size 20); Transformer used two transformer blocks, each with 4 self-attention heads (size 4) and a 1D-convolutional feed forward layer (size 16).
For all models, we used Adam as the optimizer and adopted a cosine annealing schedule to repeatedly decrease the learning rate and then restart with a higher learning rate [53], with an initial learning rate of 0.001, an annealing decay of 0.95, and an annealing step size of 100. We isolated 10% of the training datasets as a validation set. All models are trained with 200 epochs, and the best epoch was picked based on the performance on the train and validation set. In addition to balanced accuracy, we further employed ROC AUC as the main evaluation metric as it indicates the overall results with varying decision boundaries in a detection problem. 3 lists out the results of all models on each of the four datasets, and Figure 7 presents the barplot of these models' ranked average performance. There are a few noteworthy observations. First, all nine depression detection models have worse performance than that of Table 2. The best model, Chikersal et al., has an average balanced accuracy of 52.0% in the cross-dataset evaluation, compared to 58.8% in the within-dataset evaluation. This performance gap, in addition to the previously described gap between the reported results in prior work and the results on our datasets, indicates that these models do not generalize well across datasets. Since all of our evaluation experiments use the same features, the first explanation of missing other sensors' features (at the end of Section 5.2) does not play a role at this stage. Therefore, the gap between within-dataset and cross-dataset evaluation is mainly caused by the distribution shift among different populations.
Second, modern ML techniques have been developed to deal with the challenge of feature shift across domains. However, these models barely work on our datasets. Among the 15 models we investigated, CSD -Person as Domain and ERM -2D-CNN achieves the highest ROC AUC (52.3%). The results are similar to the results of traditional depression detection models (The best model Chikersal et al. achieves an ROC AUC of 54.1%.). These evaluation outcomes show that recent domain generalization methods do not work well on our datasets. This can be explained by the fact that most of these methods were developed under the context of CV or NLP tasks. Their generalizability may be affected when applied to longitudinal behavior data. Another interesting finding is the good performance of the naive ERM-based methods. Most of them (except ERM-Transformer) rank top 10 among the total of 26 methods, outperforming many generalization methods such as DANN and MLDG. Such a finding is consistent with the results in DomainBed [36] and DeepDG [94]. Both pointed out that ERM baseline often has competitive generalization performance.
Finally and most importantly, among all 26 models, our newly proposed Reorder model achieves the highest ROC AUC of 57.5% and the highest balanced accuracy of 55.2%. As shown in Figure 7, Reorder stands out among all methods. It outperforms the other models by at least 3.4% on ROC AUC (6.3% relative advantage), and 3.2% on absolute balanced accuracy (6.2% relative advantage), both with statistical significance (𝑝 < 0.05). Since Reorder has the same 1D-CNN backbone as ERM-1D-CNN, the comparison between these two models reveals the effect of adding the second reorder puzzle-solving task, which boosts the performance by 5.9% on ROC AUC (11.4% relative advantage) and 3.9% on balanced accuracy (7.6% relative advantage). Such an improvement illustrates that learning the temporal continuity of behavior trajectory can enhance the model's generalizability. However, although Reorder shows positive signals on domain generalization, it is worth noting that our model still has 0.590 0.525 0.526 0.523 0.541 0.545 0.503 0.523 0.508 0.520 + * ERM -1D-CNN [89] 0.511 0.530 0.512 0.511 0.516 0.503 0.510 0.522 0.520 0.514 + * ERM -2D-CNN [89] 0.508 0.509 0.535 0.542 0.523 0.507 0.504 0.523 0.534 0.517 + * ERM -LSTM [89] 0.518 0.516 0.536 0.511 0.520 0.511 0.502 0.528 0.502 0.511 + * ERM -Transformer [89] 0.508 0.464 0.527 0.486 0.496 0.513 0.474 0.510 0.488 0.496 -ERM -Mixup [113] 0.512 0.522 0.513 0.520 0.517 0.517 0.505 0.519 0.513 0.514 + * * * IRM [2] 0.546 0.514 0.518 0.507 0.521 0.532 0.513 0.524 0.510 0.520 + * * DANN -Dataset as Domain [32] 0.501 0.510 0.465 0.506 0.496 0.499 0.500 0.500 0.500 0.500 -DANN -Person as Domain [32] 0.512 0.511 0.470 0.507 0.500 0.500 0.500 0.500 0.500 0.500 -CSD -Dataset as Domain [76] 0.519 0.530 0.516 0.504 0.517 0.517 0.530 0.504 0.494 0.511 + * CSD -Person as Domain [76] 0.510 0.521 0.524 0.536 0.523 0.500 0.513 0.523 0.522 0.514 + * * MLDG -Dataset as Domain [50] 0.495 0.475 0.519 0.496 0.496 0.501 0.470 0.523 0.501 0.499 -MLDG -Person as Domain [50] 0.509 0.539 0.512 0.488 0.512 0.499 0.522 0.498 0.483 0.501 + MASF -Dataset as Domain [26] 0.505 0.514 0.506 0.522 0.512 0.496 0.499 0.509 0.499 0.501 + * MASF -Person as Domain [26] 0.508 0.502 0.485 0.523 0.504 0.507 0.507 0.489 0.498 0.500 + Siamese Network [47] 0.512 0.509 0.488 0.508 0.504 0.512 0.509 0.488 0.508 0.504 + Clustering 0.522 0.502 0.497 0.521 0.511 0.518 0.505 0.499 0.517 0.510 + • Reorder 0.584 0.588 0.580 0.548 0.575 0.570 0.546 0.558 0.535 0.552 + * * * great room for improvement. An ROC AUC of 57.5% is still far from being deployable in real-life scenarios, and we need more future research to improve model generalizability (see Section 6).
In addition to the leave-one-dataset-out evaluation, we conducted additional experiments to obtain more insights into the models' generalizability and investigate different generalization challenges. As the four datasets were collected from two institutes across two years, we can evaluate how these models generalize across institutes (i.e., different populations) (Section 5.4.1), and across years (i.e., different users within the same population) (Section 5.4.2). Moreover, in each institute, there were a small number of people who participated in both years. Thus, we also evaluated the models on these subsets of users across years to test generalization across the same participants at different times (Section 5.4.3). presents the ranked average balanced accuracy of all methods (train/test on institute 1/2 and then on institute 2/1). Overall, the performance is not as good compared to Figure 7. Xu et al.-Personalized has the best performance, but it only achieves an ROC AUC of 53.1%, followed by our method Reorder (52.4%). These lower performance results reflect that cross-institute generalization is a more challenging task.
5.4.2 Cross-Year. Similarly, we used two datasets from one year as the training set, and the rest from the other year as the testing set. The training setup was kept the same. The middle portion of Figure 8 shows the average balanced accuracy (train/test on year 1/2 and then on year 2/1). Compared to the cross-institute evaluation, the results of the cross-year evaluation are slightly better. Our method Reorder has the best performance, with an ROC AUC of 54.2%. This indicates that cross-year generalization could be easier than cross-institute generalization. Moreover, some models have interesting contrasting results. For example, Clustering ranks among the top 10 in the cross-institute evaluation, while it ranks among the bottom 5 in the cross-year evaluation (Δ = 2.1%). In contrast, Xu et al.-Interpretable has the worst performance in the leave-one-dataset-out evaluation, but ranks No.4 and No.9 in the cross-institute and cross-year evaluation (Δ = 4.0% and 3.9%). This means that these models capture different aspects of domain generalization.
Cross-Year with Overlapping People. We further narrowed down the evaluation to the overlapping people across years (there are no overlapping users across institutes), i.e., we used trained a model on overlapping users in one dataset, and tested the model on these users in the other dataset, with the same training details and dataset setup as Section 5.4.2. The right side of Figure 8 shows the average balanced accuracy. We observe a strong increase in performance. Reorder again achieves the best performance with an ROC AUC of 61.6%, which is 7.4% higher than the best result in Section 5.4.2. The advantage could be explained by the fact that the same users' We speculate that the comprehensive feature aggregation and selection pipeline proposed in [19] may identify more generalizable features. Overall, the cross-institute and cross-year evaluation results further illustrate more insights into model generalizability. Our model Reorder has the best or the second best results across the different tasks, revealing its advantages over other models. Moreover, the results of the third cross-dataset setup (i.e., different times of the same users) are clearly better than those of the other two setups, which reveals that the individual differences (no matter whether that is within or between populations) may play the most important role in the cross-dataset generalization challenge.
One of the major obstacles of domain generalization is the overfitting onto the training set [94,118]. We also observed a similar overfitting issue during our deep models' training process. For example, during the training, Reorder achieves an average training ROC AUC of 74.9% and a training balanced accuracy of 67.7% for the main task, as well as an average training accuracy of 30.5% for the reordering task (as a 200-class classification task). These results of the main task are much better than the ones on the testing set (note that there is no reordering task on the testing set.). As the training epoch increases, we first observe a generally increasing performance curve on the testing set (learning), and then a decreasing trend (overfitting). If we can find the "optimal" training epoch for each model via early stopping, the results could then reflect the "upper bound" of these models' performance.
We conducted such an experiment with the similar leave-one-dataset-out setup as Section 5.3. Note that this experiment was only applicable to deep-learning-based algorithms as their training process has multiple epochs. Specifically, for every model, we iterated through the training epoch from 1 to 200. We performed the same epoch selection at each epoch number based on the best performance on the train and validation set. We then compared the test performance across the 200 epochs and identified the best epoch, assuming we could stop training at this epoch. Note that this step involved a little information leakage as it leveraged the testing set to select the epoch. Thus the results only reflect the theoretical upper bound performance. When we have a large validation dataset in the future, we could potentially achieve similar results as on the test set, so that information leakage will no longer be necessary.
Figure 9 summarizes the generalizability of the deep-learning models to a new data set, when training is halted at the optimal training epoch. This figure shows that even under optimal conditions, current algorithms do not generalize well. In addition, we looked at improvement over standard training. Clustering achieves the best performance, with an average ROC AUC of 58.2% and an average balanced accuracy of 59.5%. This is a large benefit from optimal stopping (Δ equals 7.1% and 8.5%, respectively), which reveals that Clustering has an overfitting problem. Reorder achieves the second-best performance, but its performance improvement is minor (Δ < 0.5%). This indicates that there is minimal overfitting in Reorder.
We further conducted more analysis to obtain better understanding of Reorder, including additional hyperparameter tuning experiments of the reordering task's loss function (Section 5.6.1) and an ablation study of different feature types (Section 5.6.2). 5.6.1 Reordering Task Tuning. The novelty of Reorder is mainly the secondary reordering task and the new loss function of the reorder task. To further investigate how much this task influences the results, we conducted experiments with different pairs of 𝛼 (controlling the weight of the reordering task's loss) and 𝛽 (controlling the amount of reordering data during the training process), using the leave-one-dataset-out setup throughout the experiments. Figure 10a presents the best results at different 𝛼s and 𝛽s.
Overall, the performance of Reorder is stable and consistently outperforms other baseline models. For 𝛼, the model performance increases as 𝛼 increases before reaching 0.2, and then it slightly decreases with larger 𝛼. As for 𝛽, the model performance also generally increases before reaching 0.7, and then drops. Both results suggest that an appropriate weight and proportion of the reordering task is helpful, but an overemphasis on the secondary task (large weight, or large proportion) can hinder the performance of the main task. 5.6.2 Feature Ablation Study. There are four feature types used in the model training: location, physical activity, phone usage, and sleep behaviors. In addition to knowing the overall performance of using all feature types, we are also interested in the importance of each type in the Reorder model. Therefore, we further conducted a feature ablation study by removing one of the four feature types from the feature set. All models are trained under the leave-one-dataset-out setup, and Figure 10b shows the results. Unsurprisingly, removing any type of features would cause the model performance to drop. Based on ROC AUC, removing sleep-related features will cause the most significant performance drop (Δ = 3.5%), and removing location features will cause the least drop (Δ = 2.5%). Based on balanced accuracy, removing physical activity features will lead to the largest drop (Δ = 3.6%), and removing location features will lead to the least drop (Δ = 1.1%). These results suggest that compared to other feature types, location-related features could potentially be the least important for model generalizability in Reorder. This may be explained by the fact that different spatial contexts across datasets (from different institutions) can restrict on the generalizability of location features more than that of other feature types.
To summarize, Table 4 highlights the key results of these models' performance under various setups.
For prior depression detection algorithms (using traditional machine learning), the "optimal" setup (the first column) has the results when an algorithm was trained and evaluated in a single dataset via 20-fold crossvalidation. The performance differences between the optimal setup and cross-dataset setups (the right four columns) indicate that these models cannot generalize to other datasets very well.
For deep-learning-based domain generalization algorithms, the "optimal" setup has the results when a deep learning model could leverage the target dataset to select the best epoch, which is another perspective of "optimal" compared to the traditional machine learning methods 7 . The performance differences between the optimal setup and cross-dataset setups indicate that existing CV-or NLP-based domain generalization algorithms do not work well in the field of longitudinal behavior modeling. Although our new algorithms outperform these methods with statistical significance, they are still far from practical deployability and have great room for improvement.
Moreover, the comparison among the four cross-dataset setups indicates that cross-year generalization among the same users is an easier task (as reflected by the overall best performance in the last column), and that individual differences, either among the same institutes or across institutes, is more difficult and challenging for model generalization.
In this section, we first discuss the relationship between our findings and previous depression literature (Section 6.1). We then analyze the challenges of domain generalization in our longitudinal behavior data format (Section 6.2), and discuss the potential alternative of domain adaptation besides domain generalization (Section 6.3). Finally, we summarize the limitations of our work (Section 6.4).
Our analysis in Section 5.1 identifies some features that are significantly associated with depression labels across all datasets. Compared to prior work that mainly used a single dataset, our multi-dataset analysis can reveal more generalizable findings at the feature level. For example, our sleep-related features show that participants with depression tended to have shorter sleep duration and more interrupted sleep. This is aligned with prior passive sensing studies [100,101] and consistent with the criteria for diagnosing depression that sleep disturbance is a common symptom of depression [85,87]. Our phone usage-related features indicate that participants checked the phone more frequently (as indicated by the more count of and longer total duration screen unlock episodes). Such behavior was observed not only at home, but also at study places. Similar results were also observed in [80,98]. This could reflect that participants with depression had more difficulties concentrating, another common symptom of depression [35,56]. Moreover, other types of features further show that participants with depression tended to spend more time staying at home, remaining sedentary, and having fewer physical activities. These behaviors reflect a clear sign of a reduction of physical movement, which is one of the diagnosis criteria of depression [13,79]. Interestingly, our location-related features also indicate that participants with a high depression scale score had a more consistent mobility routine. They visited less uncommon places than participants without depressive symptoms, and showed a stronger repetitive pattern in their locomotion trajectories. We did not find any depression literature that is directly related to this observation. We speculate that this lack of novelty seeking could be a sign of diminished interest in other activities other than the major routines (e.g., going to class, having meals) -another common symptom of depression [3] -but it requires further analysis. Our findings may suggest some new research questions for behavioral science researchers.
We observed a large gap between the performance of prior depression detection models across our 4 datasets and their reported performance in the corresponding literature on single datasets (Section 5.2). Combining the cross-dataset evaluation results of these models in Section 5.3, we found that the observed performance gap is mainly caused by the distribution shift between users in different datasets, i.e., a feature that can be used to detect depression effectively in one dataset becomes less informative in another dataset. This observation highlights that depression detection work within a single dataset can potentially introduce dataset bias or feature distribution bias into model evaluation and conclusions. Moreover, the comparison of multiple cross-dataset setups in Section 5.4 further indicates that the individual behavior differences (both within and between populations) introduces a more significant domain generalization challenge than temporal differences of the same users (i.e., how users may change from one year to the next).
However, such a dataset difference is not easy to address. Due to the limited amount of ground truth for each individual (a common property of longitudinal behavior datasets), generalization is more challenging. Our evaluation of recent deep learning domain generalization techniques indicates that these computer vision-based methods do not work well on longitudinal passive sensing data. Although we demonstrate that building a reordering puzzle can effectively force the model to learn behavior continuity, obtain generalizable behavior feature representation, and improve the generalizability under various setups, our Reorder model still cannot fully address the challenge of feature difference across datasets. It only achieved an ROC AUC of 57.5% and a balanced accuracy of 55.2% in the leave-one-dataset-out task. The model is not robust enough for real-life deployment, and we acknowledge that there still exists great room for improvement.
Our work serves as the first step toward cross-dataset model generalizability evaluation, and we hope future studies on behavior modeling algorithms will improve the generalizability across datasets and enable field deployment. Further, we provide an open-source platform on which other researchers and developers can conduct experiments and analyses about model generalizability.
Our analysis may also suggest the need to move from domain generalization to domain adaptation, i.e., allow the model to access a small fraction of the data of new datasets or new users. As shown in our dataset analysis with the "Distinguish-The-Person" task (Figure 5b), a distribution shift exists even at the individual level. When trained on the same participants across years in both institutions, there is still a loss of performance, as reflected in the right side of Figure 8. The best Reorder model's ROC AUC is below 65%. Thus, when treating each individual as a domain, a cross-dataset generalization would mean that a model needs to generalize to tens of (or even hundreds and thousands of) new domains/individuals in a new dataset, which could be overly challenging. A more feasible way is to allow the model to adapt and personalize to each user by tuning its parameters on a small amount of data from the target individual [10,34,52]. There have been some recent promising advances in this direction [33,39]. Our analysis of optimal early stopping in Figure 9 also shows the potential of improving the model performance by accessing some data in the target dataset (using the target dataset as the test set to select the optimal epoch). If so, it provides the opportunity to leverage a wide range of transfer learning techniques in the domain adaptation area [74,95]. This idea is practical in a real-life deployment: after a model is trained on existing datasets and applied to a new user, it can first accumulate data for the first few weeks and tune the model based on the new user's behavior. Such an adaptation process is necessary if we are moving towards the next step of providing personalized intervention experience [64,108]. Although such a design would require the new user to provide extra behavior labels, it can potentially address the individual distribution shift and achieve model personalization.
There are a few limitations in our work. First, our current datasets only cover four data types and rely on RAPIDS for feature extraction. Due to the differences between the data collections of the two research groups, other data types (e.g., Bluetooth, call logs, and audio) were not included. Also, our current datasets mainly have college students as participants, which may not represent the general population. We look forward to having more research groups join our generalization effort and contribute datasets that contain more data types and sub-populations. Moreover, the current feature set on RAPIDS is still limited. In the future, there are more potential behavioral features that can be integrated into RAPIDS. A second limitation is that our Institute1Year1 dataset does not contain a weekly well-established depression scale. Therefore, in Section 3.1.1, we trained a simple decision tree using Institute1Year2 dataset to generate labels for the Institute1Year1 dataset. Although the rule obtained from the decision tree (no depression if distressed is less than 2 and nervous is less than 3) is reasonable together with an acceptable accuracy of 74.5%, the rule might introduce errors that could distort the labels and the model training. Moreover, even the clinical measurement of depression is debated, making it inherently difficult to test models [30]. In the future, researchers may consider alternative ground truths as the prediction targets. Third, the current version of our platform GLOBEM implemented a limited number of domain generalization techniques. There are more techniques that seem promising, e.g., CORAL [84], GroupDRO [81], ANDMask [75]. Moreover, our current algorithms mainly use cross entropy as the loss function, which is not designed for imbalanced datasets. Other loss functions that better deals with class imbalance, such as balanced cross entropy and focal loss [51], can be tested and evaluated on our platform. Our open-source platform was designed to be easily extensible and we plan include these algorithms in the future.
In this work, we highlight the importance of a behavior model's cross-dataset generalizability. Using depression detection as an example, we take the first step towards a systematic cross-dataset generalization evaluation in the longitudinal behavior modeling domain. We combined the efforts of two research groups across two institutes, each with two years of data, and established four datasets with a set of consistent features. We re-implemented nine prior depression detection methods, built eight recent domain generalization algorithms, and proposed two new methods for better generalizability. Our evaluation of these methods on our datasets demonstrated that existing algorithms barely outperform the baseline on cross-dataset generalization tasks, and that our new method Reorder could learn the continuity of behavior trajectories and achieve better generalizability across datasets.
Although statistically significant, its performance advantage is marginal, leaving great room for improvement. Moreover, the comparison of multiple generalization tasks indicates that individual behavior differences are the main source of challenges of domain generalization in the longitudinal behavior modeling area. We integrated all methods and open-sourced a benchmark platform named GLOBEM to assist future researchers in testing existing methods and developing new algorithms. We call for researchers in our field to pay more attention to the importance of cross-dataset generalizability evaluation. We envision this step as a necessary and essential part of behavior model deployment in the future.
Our setup of the domain generalization tasks on passive sensing data aligns with ML community's definition of generalizability as an ML model's generalization ability, that is, how well a model can adapt properly to new, previously unseen data[63,68]. This term is also closely related to one type of reproducibility of scientific research: conceptual reproducibility, the ability to obtain similar findings under new conditions that match the theoretical description of the original experiment[37,58] In this paper, we follow a few recent ML publications[1,110] and use the term generalizability. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December 2022.
Our Institute1Year1 dataset contains two quarters' data. To make it consistent with other datasets, we took the Spring quarter for our analysis. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December 2022.
For simplicity, we use "with/without depression" throughout the paper. It is worth noting that this does not imply a clinical diagnosis.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No.
4, Article 190. Publication date: December 2022.
Please refer to https://www.rapids.science/1.6/features/feature-introduction/ for features in each data type.
Due to IRB requirements, we are not able to release the full data analyzed in this article. However in[107], we release a data set that includes and extends the data from Institute 1 analyzed in this article.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol.
6, No. 4, Article 190. Publication date: December 2022.
It is worth noting that our re-implementation may not be exactly the same as the prior work due to the lack of open-source code. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December 2022.
Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December 2022. 190:14 • Xu et al.
The single dataset evaluation using deep-learning models has worse results compared to the best-epoch approach. We speculate the potential explanation is the limited sample size.Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., Vol. 6, No. 4, Article 190. Publication date: December 2022.