Figure 1: Each WILDS dataset (Koh et al., 2021) contains labeled data from the source domains (for training), validation domains (for hyperparameter selection), and target domains (for held-out evaluation). In the WILDS 2.0 update, we extend these datasets with unlabeled data from a combination of source, validation, or target domains, as well as extra domains from which there is no labeled data. The labeled data is exactly the same as in WILDS 1.0. In this figure, we illustrate the setting with the GLOBALWHEAT-WILDS dataset, where domains correspond to images acquired from different locations and at different times.

In this paper, we make two contributions. First, we present WILDS 2.0 (Figure 2), an updated version of the recent WILDS benchmark of in-the-wild distribution shifts (Koh et al., 2021). WILDS datasets span a wide range of tasks and modalities, and each dataset reflects a domain generalization or subpopulation shift setting with a substantial gap between in-distribution and out-of-distribution performance. However, WILDS 1.0 only contained labeled data, which limits the leverage for learning robust models. In WILDS 2.0, we extend 8 of the 10 WILDS datasetsfoot_0 with curated unlabeled data acquired from the same source and target domains as the labeled data, as well as from extra domains of the same type: e.g., in the GLOBALWHEAT-WILDS dataset pictured in Figure 1, we acquired unlabeled photos of wheat fields from the source and target farms as well as extra farms that were not in the original labeled dataset. In total, WILDS 2.0 adds 14.5 million unlabeled examples, expanding the number of examples for each dataset by 3-13× and allowing us to combine the real-world relevance of WILDS with the leverage of unlabeled data.

Second, we developed a standardized and consistent protocol for evaluating methods that leverage the unlabeled data in WILDS 2.0. We assessed representatives from three popular categories: methods for learning domain-invariant representations (Sun & Saenko, 2016;Ganin et al., 2016), self-training methods (Lee, 2013;Sohn et al., 2020;Xie et al., 2020), and pre-training methods that rely on self-supervision (Devlin et al., 2019;Caron et al., 2020). These methods have been successful on some types of shifts, such as going from photos to sketches, or from handwritten digits to street signs (Berthelot et al., 2021;Zhang et al., 2021).

Our results on WILDS are mixed: many methods did not outperform standard supervised training despite using additional unlabeled data, and the only clear successes were on two image classification datasets (CAMELYON17-WILDS and FMOW-WILDS). Successful methods relied heavily on data augmentation (Xie et al., 2020;Caron et al., 2020), which limited their applicability to modalities where augmentations are not as well developed, such as text and molecular graphs. The same methods were unsuccessful on image regression and detection tasks, which have been relatively understudied: e.g., pseudolabel-based methods do not straightforwardly apply to regression. For the text datasets, continued language model pre-training did not help, unlike in prior work (Gururangan et al., 2020). Our results suggest fruitful avenues for future work, such as developing data augmentations for non-image modalities and more effective hyperparameter tuning protocols.

Overall, our results underscore the importance of developing and evaluating methods for unlabeled data on a wider variety of real-world shifts than is typically studied. To this end, we have updated the open-source Python WILDS package to include unlabeled data loaders, compatible implementations of all the methods we benchmarked, and scripts to replicate all experiments in this paper (Ap- pendix G). Code and public leaderboards are available at https://wilds.stanford.edu. By allowing developers to easily test algorithms across the variety of datasets in WILDS 2.0, we hope to accelerate the development of methods that can leverage unlabeled data to improve robustness to real-world distribution shifts.

Finally, we note that WILDS 2.0 not a separate benchmark from WILDS 1.0: the labeled data and evaluation metrics are exactly the same in WILDS 1.0 and WILDS 2.0, and future results should be reported on the overall WILDS benchmark, with a note describing what kind of unlabeled data (if any) was used. In this paper, we discuss the addition of unlabeled data and analyze the performance of methods that use the unlabeled data. For a more detailed description of the datasets, evaluation metrics, and models used, please refer to the original WILDS paper (Koh et al., 2021).

WILDS 2.0 offers a diverse range of applications and modalities while also providing an extensive amount of unlabeled data that can be used as leverage for training robust models. In this section, we briefly compare with other existing ML benchmarks for unsupervised adaptation.

Images. Evaluations of unsupervised adaptation methods for image classification have focused on generalizing from natural photos to a range of stylized images, such as sketches and cartoons (PACS (Li et al., 2017), Office-Home (Venkateswara et al., 2017), and DomainNet (Peng et al., 2019)), product images (Office-31 (Saenko et al., 2010)), and synthetic renderings (VisDA (Peng et al., 2018)), though location-based shifts have also been recently explored (Dubey et al., 2021). It is also popular to evaluate on shifts between digits datasets, such as MNIST (LeCun et al., 1998), SVHN (Netzer et al., 2011), and USPS (Hull, 1994). In image detection and segmentation, existing adaptation benchmarks tend to focus on generalizing from synthetic to natural scenes (Ros et al., 2016;Richter et al., 2016;Cordts et al., 2016;Hoffman et al., 2018), which can be an important tool for realistic problems but is not the focus of this work. In contrast, WILDS considers real-world distribution shifts, and it spans diverse modalities (satellite, microscope, agriculture, and camera trap images) and tasks (classification, regression, detection).

Text. Methods for unsupervised adaptation in NLP are typically evaluated on domain shifts between different textual sources, such as news articles, different categories of product reviews, Wikipedia, or social media platforms (Blitzer et al., 2007;Mansour et al., 2009;Oren et al., 2019;Miller et al., 2020;Kamath et al., 2020;Hendrycks et al., 2020), or even more specialized sources such as legal documents (Chalkidis et al., 2020) or biomedical papers (Lee et al., 2020b;Gu et al., 2020). Multilingual tasks can also be a setting for unsupervised adaptation (Conneau et al., 2018;Conneau & Lample, 2019;Hu et al., 2020a;Clark et al., 2020), especially when generalizing to low-resource languages (Nekoto et al., 2020). The WILDS text datasets differ in that they focus on subpopulation performance, either to particular demographics in CIVILCOMMENTS-WILDS or to tail populations in AMAZON-WILDS, rather than on adapting to a completely distinct domain.

Molecules. While unlabeled molecules have been used for pre-training (Hu et al., 2020c;Rong et al., 2020), no standardized unsupervised adaptation benchmarks have been developed.

As in WILDS 1.0, we study the domain shift setting where the data is drawn from domains d ∈ D. We consider several variants of the domain shift setting. In some applications, all four types of domains are disjoint (e.g., if we are training on labeled data from some hospitals but seeking to generalize to new hospitals); in others, the target domains are a subset of the source domains (e.g., if we are training on a heterogeneous dataset but seeking to measure model performance on particular demographic subpopulations). Models are trained on labeled data from the source domains, as well as unlabeled data of one or more types of domains, depending on what is realistic for the application.

WILDS 2.0 augments 8 WILDS datasets with curated unlabeled data. For consistency, the labeled datasets and evaluation metrics are exactly the same as in WILDS 1.0, which allows direct evaluations of the utility of unlabeled training data. The labeled and unlabeled data are disjoint, e.g., the unlabeled target data is different from the labeled target data used for evaluation. Here, we briefly describe each dataset, why unlabeled data can be realistically obtained for the corresponding task, and how it might help. In Appendix A, we provide more information on each dataset, including data provenance and details on data processing. In general, all of the unlabeled datasets in WILDS 2.0 were processed in a similar way as their corresponding labeled datasets from WILDS 1.0.

IWILDCAM2020-WILDS: Species classification across different camera traps. The task is to classify the animal species in a camera trap image (Beery et al., 2020). We aim to generalize to new camera trap locations despite variations in illumination, background, and label frequencies (Beery et al., 2018). While hundreds of thousands of camera traps are active worldwide, only a small subset of these traps have had images labeled, and the unlabeled data from the other camera traps capture diverse operating conditions that can be used to learn robust models. In this work, we add unlabeled images from 3,215 extra camera traps also in the WCS Camera Traps dataset (Beery et al., 2020). This expands the number of camera traps by 11× and the number of examples by 5×.

CAMELYON17-WILDS: Tumor identification across different hospitals. The task is to classify image patches from lymph node sections as tumor or normal tissue. We seek to generalize to new hospitals, which can differ in their patient demographics and data acquisition protocols (Veta et al., 2016;AlBadawy et al., 2018;Komura & Ishikawa, 2018;Tellez et al., 2019). While obtaining labeled data for histopathology applications requires pain-staking annotations from expert pathologists, hospitals typically accumulate unlabeled slide images during normal operation. These unlabeled images could be used to adapt to differences between hospitals (e.g., different staining protocols might lead to different color distributions). We provide unlabeled patches from train and test hospitals, which expands the total number of patches by 7.5×. Both the labeled and unlabeled data are adapted from the Camelyon17 dataset (Bandi et al., 2018).

FMOW-WILDS: Land use classification across different regions and years. The task is to classify the type of building or land usage in a satellite image. Given training data from before 2013, we aim to generalize to satellite imagery taken after 2013, while maintaining high accuracy across all geographic regions. While labeling land use requires combining map data and expert annotations, unlabeled data is available in all locations in the world through constant streams of global satellite imagery. Prior work has shown that unlabeled satellite data can improve OOD accuracy in landcover and cropland prediction (Xie et al., 2021a) as well as aerial object and scene classification (Reed et al., 2021). We provide unlabeled satellite imagery across all regions from the train and test timeframes defined in WILDS, expanding the dataset by 3.5×. Both the labeled and unlabeled data are adapted from the FMoW dataset (Christie et al., 2018).

POVERTYMAP-WILDS: Poverty mapping across different countries. The task is to predict a real-valued asset wealth index of the area in a satellite image. We consider generalizing across different countries. Like FMOW-WILDS, unlabeled satellite imagery is available globally, while labeled data is expensive to collect as it requires conducting nationally representative surveys in the field. Prior work on poverty prediction has used unlabeled data for entropy minimization (Jean et al., 2018) and pre-training on auxiliary tasks such as nighttime light prediction (Xie et al., 2016;Jean et al., 2016), but these studies do not study generalization to new countries. We provide unlabeled satellite imagery from both train and test countries, expanding the dataset by 14×. Both the labeled and unlabeled data are adapted from Yeh et al. (2020).

GLOBALWHEAT-WILDS: Wheat head detection across different regions. The task is to localize wheat heads in overhead field images. We seek to generalize across image acquisition sessions, each of which represents a particular location, time, and sensor; these can differ in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination, and acquisition protocols. Wheat field images contain many densely packed and overlapping instances, making labeling wheat heads in images costly, tedious and sensitive to the individual annotator. However, hundreds of agricultural research institutes around the world collect terabytes of unlabeled field images which could be used for training. We add unlabeled field images from train, test, and extra acquisition sessions, expanding the dataset by 10×. The labeled and unlabeled data are adapted from the Global Wheat Head Detection dataset and its underlying sources (David et al., 2020;2021).

OGB-MOLPCBA: Molecular property prediction across different scaffolds. The task is to predict the biological activity of small molecules represented as molecular graphs (Wu et al., 2018;Hu et al., 2020b). We seek to generalize to molecules with new scaffold structures. Labels on biological activity are only available for a small portion of molecules, as they require expensive lab experiments to obtain. However, unlabeled molecule structures are readily available in large-scale chemical databases such as PubChem (Bolton et al., 2008), and have been previously used for pretraining (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020). We provide 5 million unlabeled molecules from source and target scaffolds, which expands the number of molecules by 12.5×. The original labeled data was curated by MoleculeNet (Wu et al., 2018) from PubChem, and we similarly extracted the unlabeled data from PubChem (Bolton et al., 2008).

CIVILCOMMENTS-WILDS: Toxicity classification across demographic identities. The task is to classify whether a text comment is toxic or not. We consider the subpopulation shift setting, where the model must classify accurately across groups of comments mentioning different demographic identities. While labels require large-scale crowdsourcing annotations on both comment toxicity, unlabeled article comments are widely available on the internet. We provide unannotated comments as unlabeled data, which expands the size of the dataset by 4.5×. Both the labeled and unlabeled data are adapted from Borkan et al. (2019).

AMAZON-WILDS: Sentiment classification across different users. The task is to classify the star ratings of Amazon reviews. We seek to perform consistently well across new reviewers. While the labels (star ratings) are always available for Amazon reviews in practice, unlabeled data is a common source of leverage for sentiment classification more generally, with prior work in domain adaptation (Blitzer & Pereira, 2007;Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009;Li et al., 2011). We provide unlabeled reviews from test and extra reviewers, which expands the total number of reviews by 7.5×. Both the labeled and unlabeled data are adapted from the Amazon review dataset by Ni et al. (2019).

For our evaluation, we selected representative methods from the three categories described below. These methods exemplify current approaches to using unlabeled data to improve robustness, and they have been successful on popular domain adaptation benchmarks like DomainNet (Peng et al., 2019) and semi-supervised settings like improving ImageNet accuracy by leveraging unlabeled images from the internet (Xie et al., 2020;Caron et al., 2020). For more details, see Appendix B.

Domain-invariant methods. Domain-invariant methods learn feature representations that are invariant across different domains by penalizing differences between learned source and target representations (Long et al., 2015;Ganin et al., 2016;Sun & Saenko, 2016;Long et al., 2017;2018;Saito et al., 2018;Zhang et al., 2018;Xu et al., 2019;Zhang et al., 2019b). We discuss these methods further in Appendix B.2. For our experiments, we evaluate two classical methods:

• Domain-Adversarial Neural Networks (DANN) (Ganin et al., 2016) penalize representations on which an auxiliary classifier can easily discriminate between source and target examples.

• Correlation Alignment (CORAL) (Sun et al., 2016;Sun & Saenko, 2016) penalizes differences between the means and covariances of the source and target feature distributions.

Self-training. Self-training methods "pseudo-label" unlabeled examples with the model's own predictions and then train on them as if they were labeled examples. These methods often also use consistency regularization, which encourages the model to make consistent predictions on augmented views of unlabeled examples (Sohn et al., 2020;Xie et al., 2020;Berthelot et al., 2021). Selftraining methods have recently been successfully applied to unsupervised adaptation (Saito et al., 2017;Berthelot et al., 2021;Zhang et al., 2021). We include three representative algorithms:

• Pseudo-Label (Lee, 2013) dynamically generates pseudolabels and updates the model each batch.

• FixMatch (Sohn et al., 2020) adds consistency regularization on top of the Pseudo-Label algorithm. Specifically, it generates pseudolabels on a weakly augmented view of the unlabeled data, and then minimizes the loss of the model's prediction on a strongly augmented view.

• Noisy Student (Xie et al., 2020) leverages weak and strong augmentations like FixMatch, but instead of dynamically generating pseudolabels for each batch, it alternates between a few teacher phases, where it generates pseudolabels, and student phases, where it trains to convergence on the (pseudo)labeled data.

Self-supervision. Self-supervised methods learn useful representations by training on unlabeled data via auxiliary proxy tasks. Common approaches include reconstruction tasks (Vincent et al., 2008;Erhan et al., 2010;Devlin et al., 2019;Gidaris et al., 2018;Lewis et al., 2020), and contrastive learning (He et al., 2020;Chen et al., 2020b;Caron et al., 2020;Radford et al., 2021b), and recent work has shown that self-supervised methods can reduce dependence on spurious correlations and improve performance on domain adaptation tasks (Wang et al., 2021;Tsai et al., 2021;Mishra et al., 2021). We use these self-supervision methods for unsupervised adaptation by first pre-training models on the unlabeled data, and then finetuning them on the labeled source data (Shen et al., 2021). We evaluate popular self-supervised methods for vision and language:

• SwAV (Caron et al., 2020) is a contrastive learning algorithm that maps representations to a set of clusters and then enforces similarity between cluster assignments.

• Masked language modeling (MLM) (Devlin et al., 2019) randomly masks some of the tokens from input text and trains the model to predict the missing tokens.

To evaluate how well existing methods can leverage unlabeled data to be robust to in-the-wild distribution shifts, we benchmarked the methods above on all applicable WILDS 2.0 datasets.

We used the default models, labeled training and test sets, and evaluation metrics from WILDS.

Unlabeled data. WILDS 2.0 contains multiple types of unlabeled data (from source, extra, validation, and/or target domains). For simplicity, we ran experiments on a single type of unlabeled data for each dataset. Where possible, we used unlabeled target data to allow methods to directly adapt to the target distribution; for IWILDCAM2020-WILDS and CIVILCOMMENTS-WILDS, which do not have unlabeled target data, we used the extra domains instead. All methods use exactly the same sets of labeled and unlabeled training data (except ERM, which does not use unlabeled data).

Hyperparameters. We tuned each method on each dataset separately using random hyperparameter search. Following WILDS 1.0, we used the labeled out-of-distribution (OOD) validation set to select hyperparameters and for early stopping (Koh et al., 2021). This validation set is drawn from a different distribution than both the training and the OOD test set, so tuning on it does not leak information on the test distribution. We did not use the in-distribution (ID) validation set. For image classification and regression, we used both RandAugment (Cubuk et al., 2020) and Cutout (DeVries & Taylor, 2017) as data augmentation for all methods. We did not use data augmentation for the remaining datasets. For some datasets, we also had ground truth labels for the "unlabeled" data, which we used to run fully-labeled ERM experiments. Overall, we ran 600+ experiments for 7,000 GPU hours on NVIDIA V100s. See Appendix B for a discussion of which methods were applicable to which datasets; Appendix C for augmentation details; Appendix F for the fully-labeled experiments; Appendix D for further experimental details.

Table 2 shows mixed results on WILDS: most methods do not improve over standard empirical risk minimization (ERM) despite access to unlabeled data and careful hyperparameter tuning. In contrast, these methods have been shown to perform well on prior unsupervised adaptation benchmarks;

in Appendix E, we verify our implementations by showing that these methods (with the exception of CORAL) outperform ERM on the real → sketch shift in DomainNet, a standard unsupervised adaptation benchmark for object classification (Peng et al., 2019).

Image classification (IWILDCAM2020-WILDS, CAMELYON17-WILDS, and FMOW-WILDS).

Data augmentation improved OOD performance on all three image classification datasets. The gain was the most substantial on CAMELYON17-WILDS, where vanilla ERM achieved 70.8% accuracy, while ERM with data augmentation achieved 82.0% accuracy.foot_1

On CAMELYON17-WILDS and FMOW-WILDS, where we had access to unlabeled target data, Noisy Student and SwAV pre-training consistently improved OOD performance and reduced variability across replicates. However, the other methods-CORAL, DANN, Pseudo-Label, and FixMatchunderperformed ERM. This was especially surprising for FixMatch, which performed very well on DomainNet (Appendix E). Both FixMatch and Noisy Student use pseudo-labeling and consistency regularization, but FixMatch dynamically computes pseudo-labels in each batch from the start of training, whereas Noisy Student first trains a teacher model to convergence on the labeled data and updates pseudolabels at a much slower rate. As in Xie et al. (2020), this suggests that dynamically updating pseudo-labels might hurt generalization.

On IWILDCAM2020-WILDS, where we had access to 4× as many unlabeled images from extra domains (distinct camera traps) but not to any images from the target domains, none of the benchmarked methods improved OOD performance compared to ERM. This was surprising, as many of these methods were originally shown to work in semi-supervised settings. One difference could be that the labeled and unlabeled examples in IWILDCAM2020-WILDS differ more significantly (as Table 2: The in-distribution (ID) and out-of-distribution (OOD) performance of each method on each applicable dataset. Following WILDS 1.0, we ran 3-10 replicates (random seeds) for each cell, depending on the dataset. We report the standard deviation across replicates in parentheses; the standard error (of the mean) is lower by the square root of the number of replicates. Fully-labeled experiments use ground truth labels on the "unlabeled" data. We bold the highest non-fully-labeled OOD performance numbers as well as others where the standard error is within range. Below each dataset name, we report the type of unlabeled data and metric used.

they originate from different camera traps) than in the original FixMatch paper (Sohn et al., 2020), which used i.i.d. labeled and unlabeled data, or the Noisy Student paper (Xie et al., 2020), which used ImageNet labeled data (Russakovsky et al., 2015) and JFT unlabeled data (Hinton et al., 2015).

Fully-labeled ERM models that used ground truth labels for the "unlabeled" data were available for FMOW-WILDS and IWILDCAM2020-WILDS. They significantly outperformed other methods, suggesting room for improvement in how we leverage the unlabeled data.

Data augmentation had no effect on performance on POVERTYMAP-WILDS, which differs from the above image datasets in that it is a regression task and involves multi-spectral satellite images (with 7 channels); both of these aspects are relatively unstudied compared to standard RGB image classification. All applicable methods underperformed standard ERM, despite having access to unlabeled data from the target domains (countries).

Image detection (GLOBALWHEAT-WILDS). We did not apply data augmentation here, as standard augmentation changes the labels (e.g., cropping the image might remove bounding boxes) and would violate the assumption that labels are invariant under augmentations, which contrastive and consistency regularization methods like SwAV, Noisy Student, and FixMatch rely on. Accordingly, we did not evaluate FixMatch and SwAV, and we modified Noisy Student to remove data augmentation noise. All applicable methods underperformed ERM.

Molecule classification (OGB-MOLPCBA). We did not apply data augmentation techniques to OGB-MOLPCBA as they are not well-developed for molecular graphs. All methods underperformed ERM. We did not report ID results as this dataset has no separate ID test set.

Text classification (CIVILCOMMENTS-WILDS, AMAZON-WILDS). Likewise, we did not apply data augmentation to the text datasets. On both datasets, other methods performed similarly to ERM (with class-balancing for CIVILCOMMENTS-WILDS). Continued masked LM pre-training on the unlabeled data did not improve target performance, unlike in prior work (Gururangan et al., 2020); this might be because the BERT pre-training corpus (Devlin et al., 2019;Hendrycks et al., 2020) is more similar to the online comments in CIVILCOMMENTS-WILDS and product reviews in AMAZON-WILDS than to the biomedical/CS text studied in Gururangan et al. (2020). Also, CIVILCOMMENTS-WILDS and AMAZON-WILDS measure subpopulation performance (on minority demographics and on the tail subpopulation, respectively), whereas prior work adapted models to new areas of the input space (e.g., from news to biomedical articles). Fully-labeled ERM models showed modest gains compared to FMOW-WILDS and IWILDCAM2020-WILDS. As our evaluations on these text datasets focus on subpopulations performance, these results are consistent with prior observations that ERM models can have poor subpopulation performance even with large labeled training sets (Sagawa et al., 2020), necessitating other approaches to subpopulation shifts.

We conclude by discussing several takeaways and promising directions for future work.

The role of data augmentation. Many unsupervised adaptation methods rely strongly on data augmentation for consistency regularization or contrastive learning. This reliance on data augmentation techniques-which are largely image-specific-restricts their generality, as they do not readily generalize to other modalities (or even other types of images besides photos). Developing data augmentation techniques that can work well in other applications and modalities could be crucial for expanding the applicability of these methods (Verma et al., 2021).

Hyperparameter tuning. Unsupervised adaptation methods have even more hyperparameters than standard supervised methods, and consistent with prior work, we found that these hyperparameters can significantly affect OOD performance (Saito et al., 2021). Moreover, unlike in standard i.i.d. settings, we do not have labeled target data that we can use for hyperparameter selection. Improved methods for hyperparameter tuning could significantly improve OOD performance. Such methods might make use of the unlabeled target data, or even the combination of labeled and unlabeled OOD validation data, which is provided for most datasets in WILDS 2.0.

Pre-training on broader unlabeled data. Pre-training on huge amounts of unlabeled data improves robustness to distribution shifts in some settings (Bommasani et al., 2021). The unlabeled data need not be related to the task: e.g., CLIP was pre-trained on text-image pairs from the internet but tested on tasks including histopathology and satellite image classification (Radford et al., 2021a). Existing techniques for this type of broad pre-training appear insufficient for WILDS: many of our models were initialized with ImageNet-pretrained weights or derivatives of BERT, but do not generalize well OOD. While we focused on providing curated unlabeled data that is closely tailored to the task, it could be fruitful to use both broad and curated unlabeled data.

Leveraging domain annotations and task-specific structure. OOD robustness is ill-posed in general, as models cannot be robust to arbitrary distribution shifts. Beyond unlabeled data, WILDS also has domain annotations and other structured metadata for both labeled and unlabeled data (e.g., in IWILDCAM2020-WILDS, we know which images were taken from which cameras). Exploiting this type of fine-grained domain structure for unsupervised adaptation-e.g., through multisource/multi-target domain adaptation methods (Zhao et al., 2018;Peng et al., 2019)-could be a promising avenue for learning models that are more robust to the domain shifts in WILDS.

All WILDS datasets are curated and adapted from public data sources, with licenses that allow for public release. The datasets are all anonymized.

The distribution shifts in several of the WILDS datasets deal with issues of discrimination and bias that arise in real-world applications. For example, CIVILCOMMENTS-WILDS studies disparate model performance across online comments that mention different demographic groups, while FMOW-WILDS and POVERTYMAP-WILDS study countries and regions where labeled satellite data is less readily available. As our results suggest, standard models trained on these datasets will not perform well on those subpopulations, and their learned representations might also be biased in undesirable ways (Bolukbasi et al., 2016;Caliskan et al., 2017;Garg et al., 2018;Tan & Celis, 2019;Steed & Caliskan, 2021). We also encourage caution in interpreting positive results on these datasets, as our evaluation metrics might not encompass all relevant facets of discrimination and bias: e.g., the "ground truth" toxicity annotations in CIVILCOMMENTS-WILDS can themselves be biased, and the particular choice of regions in FMOW-WILDS might obscure lower model performance in sub-regions.

For FMOW-WILDS and POVERTYMAP-WILDS, surveillance and privacy issues also need to be considered. In FMOW-WILDS, the image resolution is lower than that of other public satellite data (e.g., from Google Maps), and in POVERTYMAP-WILDS, the location metadata is noised to protect privacy. For a deeper discussion of the ethics of remote sensing in the context of humanitarian aid and development, we refer readers to the UNICEF report by Berman et al. (2018).

All WILDS datasets are publicly available at https://wilds.stanford.edu, together with code and scripts to replicate all of the experiments in this paper. We also provide all trained model checkpoints and results, together with the exact hyperparameters used.

In our appendices, we provide more details on the datasets and experiments:

• In Appendix A, we describe each of the updated datasets in WILDS 2.0 and their sources of unlabeled data as well as what data processing steps were taken.

• In Appendix B, we describe the implementations of each of our benchmarked methods in detail. In particular, we discuss any changes we made to their original implementations, either for consistency with other methods or with prior implementations of these methods.

• In Appendix C, we describe details of the data augmentations (if any) that we used across each dataset.

• In Appendix D, we describe our experimental protocol, including the hyperparameter selection procedure and hyperparameter grids for all of the methods and datasets.

• In Appendix E, we describe the details of our experiments on DomainNet.

• In Appendix F, we describe the details of our fully-labeled ERM experiments.

• Finally, in Appendix G, we include an illustrative code snippet of how to use the data loaders in the WILDS library.

The project was initiated by Shiori Sagawa, Pang Wei Koh, and Percy Liang. Shiori Sagawa and Pang Wei Koh led the project and coordinated the activities below. Tony Lee developed the experimental infrastructure and ran the experiments. Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, and Michihiro Yasunaga designed the evaluation framework and implemented the algorithms. The unlabeled data loaders and corresponding dataset writeups were added by:

• AMAZON-WILDS: Tony Lee

• CAMELYON17-WILDS: Tony Lee

In this appendix, we provide additional details on the unlabeled data in WILDS 2.0. For more context on the motivation behind each dataset, the choice of evaluation metric, and the labeled data, please refer to the original WILDS paper (Koh et al., 2021).

A.1 IWILDCAM2020-WILDS

The IWILDCAM2020-WILDS dataset was adapted from the iWildCam 2020 competition dataset made up of data provided by the Wildlife Conservation Society (WCS) (Beery et al., 2020) Broader context. There are large volumes of unlabeled natural world data that have been collected in growing repositories such as iNaturalist (Nugent, 2018), Wildlife Insights (Ahumada et al., 2020), and GBIF (Robertson et al., 2014). This data includes images or video collected by remote sensors or community scientists, GPS track data from an-animal devices, aerial data from drones or satellites, underwater sonar, bioacoustics, and eDNA. Methods that can harness the wealth of information in unlabeled ecological data are well-posed to make significant breakthroughs in how we think about ecological and conservation-focused research. Natural-world and ecological benchmarks that provide unlabeled data include NEWT (Van Horn et al., 2021), investigating efficient task learning, and Semi-Supervised iNat (Su & Maji, 2021), which provides labeled data for only a subset of the taxonomic tree. Recent work has begun to adapt weakly-supervised and self-supervised approaches for these natural world settings, including probing the generality and efficacy of self-supervision (Cole et al., 2021), incorporating domain-relevant context into self-supervision (Pantazis et al., 2021), or leveraging weak supervision from alternative data modalities (Weinstein et al., 2019) or pre-trained, generic models (Weinstein et al., 2021;Beery et al., 2019). Active learning also plays a role here in seeking to adapt models efficiently to unlabeled data from novel regions with only a few targeted labels (Kellenberger et al., 2019;Norouzzadeh et al., 2021).

The CAMELYON17-WILDS dataset (Koh et al., 2021) was adapted from the Camelyon17 dataset (Bandi et al., 2018), which is a collection of whole-slide images (WSIs) of breast cancer metastases in lymph node sections from 5 hospitals in the Netherlands. The labels were obtained by asking expert pathologists to perform pixel-level annotations of each WSI, which is an expensive and painstaking process. In practice, unlabeled WSIs (i.e., WSIs without pixel-level annotations) are much easier to obtain. For example, only a fraction of the WSIs in the original Camelyon17 dataset (Bandi et al., 2018) were labeled; the other WSIs, which are taken from the same 5 hospitals, were provided without labels. In this work, we augment the CAMELYON17-WILDS dataset with unlabeled data from these WSIs.

Problem setting. The task is to classify whether a histological image patch contains any tumor tissue. We consider generalizing from a set of training hospitals to new hospitals at test time. The input x corresponds to a 96×96 image patch extracted from an WSI of a lymph node section, the label y is a binary indicator of whether the central 32×32 patch of the input contains any pixel that was annotated as a tumor in the WSI, and the domain d identifies which hospital the patch came from. Each patch also includes metadata on which WSI it was extracted from, though we do not use this metadata for training or evaluation. Models are evaluated by their average accuracy on a class-balanced test dataset.

Data. All of the labeled and unlabeled data are taken from the Camelyon17 dataset (Bandi et al., 2018), which consists of WSIs from 5 hospitals (domains) in the Netherlands. We provide unlabeled data from same domains as the labeled CAMELYON17-WILDS dataset (no extra domains). The domains are split as follows:

1. Source: Hospitals 1, 2, and 3.

2. Validation (OOD): Hospital 4.

3. Target (OOD): Hospital 5.

CAMELYON17-WILDS also includes a Validation (ID) set which contains data from the training hospitals.

The CAMELYON17-WILDS dataset has a total of 455,954 labeled patches across these splits, derived from the 10 WSIs per hospital that have full pixel-level annotations. We augment the dataset with a total of 2,999,307 unlabeled patches, extracted from an additional 90 unlabeled WSIs per hospital.

There is no overlap between the WSIs used for the labeled versus unlabeled data. To extract and process each patch, we followed the same data processing steps that were carried out for the labeled data in Koh et al. (2021).

Unlike the labeled patches, which were sampled in a class-balanced manner (i.e., half of the patches have positive labels), we sampled the unlabeled patches uniformly at random from the unlabeled WSIs. We sampled 6,667 patches per unlabeled WSI, with the single exception of one WSI which had only 5,824 valid patches, resulting in a total of 3,000,150 unlabeled patches (Table 4). While the labeled patches were sampled in a class-balanced manner, the underlying label distribution skews heavily negative (approximately 95% of the patches in a WSI are negative), so we expect the unlabeled patches to be similarly skewed in their label distribution.

Broader context. We focused on providing unlabeled data from the same hospitals (domains) as in the original labeled CAMELYON17-WILDS dataset. This unlabeled data from the training and test hospitals can be used to develop and evaluate methods for semi-supervised learning (Peikari et al., 2018;Akram et al., 2018;Lu et al., 2019;Shaw et al., 2020) and domain adaptation (Ren et al., 2018;Zhang et al., 2019a;Koohbanani et al., 2021), respectively. In practice, there is also a large amount of unlabeled data from different domains that is publicly available: for example, The Cancer Genome Atlas (TCGA) hosts tens of thousands of publicly-available slide images across a variety of cancer types and from many different hospitals (Weinstein et al., 2013). These large and diverse datasets need not even be directly relevant to the task at hand, e.g., one could pretrain a model on images for different types of cancer even if the goal were to develop a model for breast cancer. Recent work has started to explore the use of these large and diverse datasets for computational pathology applications (Ciga et al., 2020;Dehaene et al., 2020) and in other medical imaging applications (Azizi et al., 2021).

The FMOW-WILDS dataset (Koh et al., 2021) was adapted from the FMoW dataset (Christie et al., 2018), which consists of global satellite images from 2002-2018, labeled with the functional purpose of the buildings or land in the image. The labels are collected by a process which combines map data with crowdsourced annotations (from a trusted crowd). In contrast, unlabeled satellite imagery is readily available across the globe. In this work, we augment the FMOW-WILDS dataset with unused satellite images that were part of the original FMoW dataset but not in the FMOW-WILDS dataset.

Problem setting. The task is to classify the building or land-use type of a satellite image. We consider generalizing from images before 2013 to after 2013, as well as considering the performance on the worst-case geographic region (Africa, the Americas, Oceania, Asia, or Europe). The input x is an RGB satellite image (224×224 pixels). The label y is one of 62 building or land use categories.

The domain d represents both the year and the geographical region of the image. Each image also includes metadata on the location and time of the image, although we do not use these except for splitting the domains. Models are evaluated by their average and worst-region accuracies in the OOD timeframe.

Data. The labeled and unlabeled data are taken from the FMoW dataset (Christie et al., 2018). We provide unlabeled data from same domains as the labeled FMOW-WILDS dataset (no extra domains). The domains are as follows:

1. Source: The FMOW-WILDS dataset has 141,696 labeled images across these splits. We augment the dataset with 340,469 unlabeled images. These images come from two sources:

1. We use a sequestered split of the dataset, which consists of new locations that are not in the original labeled FMOW-WILDS dataset; these unlabeled data are drawn from the same distribution as the labeled data.

2. For the unlabeled target and validation splits, we also add unlabeled data in their respective timeframes from the training set locations. While the unlabeled data from the Validation (OOD) and Target (OOD) domains can come from the same locations as the labeled training data, we note that none of the locations in the labeled Validation (OOD) or Target (OOD) data, which is used for evaluation, is shared with any of the unlabeled or labeled data used for training.

Broader context. We focus on providing unlabeled data from the years (domains) that were in the original FMOW-WILDS dataset. Prior works have used unlabeled satellite imagery for pretraining (Xie et al., 2016;Jean et al., 2016;Xie et al., 2021a;Reed et al., 2021), self-training (Xie et al., 2021a), and semi-supervised learning (Reed et al., 2021). Leveraging unlabeled satellite imagery is powerful since it is widely available and can reduce the frequency at which we need to re-collect labeled data.

The POVERTYMAP-WILDS dataset (Koh et al., 2021) was adapted from Yeh et al. (2020). The dataset consists of satellite images from 23 African countries, labeled with a village-level realvalued asset wealth index (measure of wealth). The labels are collected by conducting a nationally representative survey, which requires sending workers into the field to ask each household a number of questions and can be very expensive. In contrast, unlabeled satellite imagery is readily available across the globe. In this work, we augment the POVERTYMAP-WILDS dataset with satellite images from the same LandSat satellite.

Problem setting. The task is to predict a real-valued asset wealth index from a satellite image. We consider generalizing across country borders (the dataset contains 5 different cross validation folds, each splitting the countries differently). The input x is a multispectral LandSat satellite image with 8 channels (resized to 224 × 224 pixels). The output y is a real-valued asset wealth index. The domain d represents the country the image was taken in, as well as whether the image was taken at an urban or rural area. Each image also includes metadata on the location and time, although we do not make use of these except for defining the domains. Models are evaluated by the average Pearson correlation (r) across 5 folds, as well as the lower of the subpopulations to test generalization to these subpopulations. In particular, generalization to rural subpopulations is important as poverty is more common in rural areas.

Data. We provide unlabeled data from same domains as the labeled POVERTYMAP-WILDS dataset (no extra domains). The domains are split as follows:

1. Source: Images from training countries in the fold.

Images from validation countries in the fold.

Images from test countries in the fold.

All the countries in these splits are disjoint. Folds also contain a Validation (ID) and Target (ID) set with data from the training countries.

The POVERTYMAP-WILDS dataset has 19,669 labeled images across these splits. We augment the dataset with 261,396 unlabeled images from the same 23 countries. These images are collected using the same process as Yeh et al. (2020) from the same LandSat satellite. The image locations are chosen to be roughly near survey locations from the Demographic and Health Surveys (DHS).

Broader context. We focus on providing unlabeled data from the countries (domains) that were in the original POVERTYMAP-WILDS dataset. Prior works on poverty prediction have used pretraining on unlabeled data (to predict an auxiliary task such as nighttime light prediction) (Xie et al., 2016;Jean et al., 2016) and for semi-supervised learning via entropy minimization (Jean et al., 2018). However, these works focus on generalization to new locations in the countries in the training set. Poverty prediction is different from usual tasks in that the output is real-valued. Most methods for unlabeled data are made for classification tasks, and we hope that our dataset will encourage more work on methods for using unlabeled data for improving OOD performance in regression tasks.

The GLOBALWHEAT-WILDS dataset was extended from the Global Wheat Head Dataset developed by David et al. (2020;2021). The goal of the dataset is to localize wheat heads from field images to assist plant scientists to assess the density, size, and health of wheat heads in a particular wheat field. This imagery is acquired during different periods to cover the development of the vegetation, from the emergence to organ appearance. Examples in GLOBALWHEAT-WILDS are labeled by bounding box annotations of each wheat head in the image. Wheat heads are densely packed and overlapping, making object annotation highly tedious. Thus, the Global Wheat Head Dataset (GWHD) is relatively small, while in reality more field images are available. We supplement GLOBALWHEAT-WILDS with unlabeled examples from the same set of field vehicles and sensors but taken in different acquisition sessions, i.e., at different locations or the same location in a different year. The inclusion of this unlabeled data allows: 1) a much higher spatial coverage of a field location when the data comes from an acquisition session which is already included, 2) a much higher temporal resolution when the data comes from a location which is already included, so we have a larger range of wheat growth stages, and 3) slightly more diversity when the session comes from a different location, but with the same image acquisition protocol (i.e., the same field vehicle and image sensor).

Problem setting. The task is to localize wheat heads in high resolution overhead field images taken from above the crop canopy. We consider generalizing across acquisition sessions representing a particular location, time and sensor with which the images were captured. Variation across sessions includes changes in wheat genotype, wheat head appearance, growing conditions, background appearance, illumination and acquisition protocol. The input x is an overhead outdoor image of wheat canopy, and the label y is a set of box coordinates bounding the wheat heads (the spike at the top of the wheat plant holding grain), omitting any hair-like awns that may extend from the head. The domain d designates an acquisition session, which corresponds to a certain location, time, and imaging sensor.

Data. We provide unlabeled data from same domains as the labeled GLOBALWHEAT-WILDS dataset. Additionally, we provide unlabeled data from extra acquisition sessions not in the labeled GLOBALWHEAT-WILDS dataset (extra domains). The domains are split as follows:

1. Source: 18 acquisition sessions in Europe (France ×13, Norway ×2, Switzerland, United Kingdom, Belgium).

2. Validation (OOD): 8 acquisition sessions: 7 in Asia (Japan × 4, China × 3) and 1 in Africa (Sudan).

3. Target (OOD): 21 acquisition sessions: 11 in Australia and 10 in North America (USA × 6, Mexico × 3, Canada).

4. Extra (OOD): 53 acquisition sessions distributed across the world.

The source, validation, and target sessions are split by continent, while the extra sessions are taken from across the world. For acquisition sessions with both labeled and unlabeled data, we randomly selected new patches of 1024x1024 pixels from the original underlying data. The images were preprocessed in the same way as described in David et al. (2021).

Broader context. Utilizing unlabeled data is relatively new in the context of plant phenotyping, due to the lack of a large, unlabeled database of plant images. However, larger plant image datasets are starting become available, such as from the Terraphenotying Reference Platform (TERRA-Ref, Burnette et al. (2018)). Increasing the sample size and variation within plant datasets is an important goal, because plants from the same species are fairly self-similar within the same field and therefore increasing the number of locations, times and image types included in a dataset can be beneficial for making fine-grained visual classifications for plants. Further, for plant phenotyping to be used in farming applications, such as for precisely spraying weeds in a field with herbicide, models must be highly robust to variations between different fields.

The OGB-MOLPCBA dataset was adapted from the Open Graph Benchmark (Hu et al., 2020b) and originally curated by the MoleculeNet (Wu et al., 2018) from the PubChem database (Bolton et al., 2008). The dataset is a collection of molecules annotated with 128 kinds of binary labels indicating the outcome of different biological assays. Performing biological assays is expensive, and as a result, the assay labels are only sparsely available over a tiny portion of the molecules curated in the large-scale PubChem database (Bolton et al., 2008). On the other hand, unlabeled molecule data is abundant and readily available from the database. Prior work in graph machine learning has leveraged unlabeled molecules to perform pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020). In this work, we augment the OGB-MOLPCBA dataset with unlabeled molecules subsampled from the PubChem database.

Problem setting. The task is multi-task molecule classification, and we consider generalizing to new molecule scaffold structures at test time. The input x corresponds to a molecular graph (where nodes are atoms and edges are chemical bounds), the label y is a 128-dimensional binary vector, representing the binary outcomes of the biological assay results. y could contain NaN values, indicating that the corresponding biological assays were not performed on the given molecule. The domain d indicates the scaffold group a molecule belongs to. As the binary labels are highly-skewed, the model's classification performance is evaluated using the Average Precision.

Data. All of the labeled and unlabeled data are taken from the PubChem database (Bolton et al., 2008). We provide unlabeled data from same domains as the labeled OGB-MOLPCBA dataset (no extra domains). We curate the unlabeled data by randomly sampling 5 million molecules from the PubChem database. We then assign these unlabeled molecules to the existing labeled scaffold groups that contain the most similar molecules. Specifically, we first compute the 1024-dimensional Morgan fingerprints for all the molecules (Rogers & Hahn, 2010;Landrum et al., 2006). Then, for each unlabeled molecule, we compute its Jaccard similarity against all the labeled molecules in OGB-MOLPCBA and obtain a labeled molecule with the highest Jaccard similarity. Finally, we assign the unlabeled molecule to the scaffold group that the most similar labeled molecule belongs to. This way, the molecules within the same scaffold groups are structurally similar to each other.

The domains in the OGB-MOLPCBA dataset are as follows:

1. Source: 44,930 scaffold groups.

2. Validation (OOD): 31,361 scaffold groups.

3. Target (OOD): 43,793 scaffold groups.

The largest scaffolds are in the source split and the smallest scaffolds in the target split. We assign all of the unlabeled molecules to the existing domains, so there are no extra domains added.

While the unlabeled data are similar to the labeled data in that they were all derived from PubChem (Bolton et al., 2008), it is quite possible that there was some selection bias in which molecules in PubChem were chosen to be labeled, which would lead to an undocumented distribution shift between the unlabeled and labeled datasets.

Broader context. We focused on providing unlabeled data for both training and OOD test domains. Unlabeled molecules can be used to develop and evaluate methods for domain adaptation, self-training, as well as pre-training (Hu et al., 2020c) and semi-supervised learning (Sun et al., 2020). In terms of future directions, we think it is fruitful to explore both graph-agnostic methods (e.g., pseudo-label training) and more graph-specific methods (e.g., self-supervised learning of graph neural networks (Xie et al., 2021b)).

The CIVILCOMMENTS-WILDS dataset (Koh et al., 2021) was adapted from the CivilComments dataset (Borkan et al., 2019), which is a collection of text comments made on online articles. The data in CIVILCOMMENTS-WILDS underwent a significant labeling and annotation process: each example was labeled toxic or non-toxic and annotated for whether they mentioned certain demographic identities by at least 10 crowdworkers. Such a substantial labeling and identity annotation process is expensive and time-consuming. On the other hand, unlabeled, unannotated text comments are readily available. For example, CIVILCOMMENTS-WILDS only contains a subset of all data available in the original CivilComments dataset (Borkan et al., 2019), most of which Koh et al. ( 2021) excluded because these examples were not annotated for mentioning identities. In this work, we augment the CIVILCOMMENTS-WILDS dataset with these unlabeled, unannotated comments.

Problem setting. The task is to classify whether a text comment is toxic or not. The input x is a text comment (at least one sentence long) originally made on an online article, the label y is a binary indicator of whether the comment is rated toxic or not, and the domain d is an 8-dimensional binary vector, where each dimension corresponds to whether the comment mentions each of 8 demographic identities: male, female, LGBTQ, Christian, Muslim, other religions, Black, or White, respectively. Each comment also includes metadata on which article the comment was made on, although we do not use this metadata for training or evaluation.

We consider the subpopulation shift setting, where the model must perform well across all subpopulations, which are defined based on d. Koh et al. ( 2021) define 16 subpopulations (groups) based on d. Models are then evaluated by their worst-group accuracy, i.e., the lowest accuracy over the 16 groups considered. In our work, we use the same evaluation setup.

Data. All of the labeled and unlabeled data are taken from the CivilComments dataset (Borkan et al., 2019). After preprocessing, Koh et al. ( 2021) created the CIVILCOMMENTS-WILDS dataset using the 448,000 examples that were fully annotated for both toxicity y and the mention of demographic identities d. In this work, we augment CIVILCOMMENTS-WILDS with an additional 1,551,515 examples collected by Borkan et al. (2019). We use these examples as unlabeled data.

We follow the same preprocessing steps as was done with the labeled data in Koh et al. (2021) 11). In practice, these comments may actually mention any number of identities.

A substantial amount (1,427,848 or 92%) of the unlabeled comments are drawn from the same articles as the labeled comments. In particular, 140,082 unlabeled comments are from the same articles as labeled comments in the test split.

CIVILCOMMENTS-WILDS exhibits class imbalance. We account for this when benchmarking methods by sampling class-balanced batches of labeled data when applicable (see Appendix B).

Broader context. In this work, we focused on supplementing CIVILCOMMENTS-WILDS with extra unannotated data from the original CivilComments dataset (Borkan et al., 2019) papers have pointed out biases in large language models (Abid et al., 2021;Nadeem et al., 2020;Gehman et al., 2020). However, recent work has also suggested that pre-trained models can be trained to be more robust against some types of spurious correlations (Hendrycks et al., 2020;Tu et al., 2020) and that additional domain-and task-specific pre-training (Gururangan et al., 2020) can also improve performance. We hope our contributions to the CIVILCOMMENTS-WILDS dataset can encourage future study on whether unlabeled data can be leveraged to improve generalization across subpopulation shifts.

The AMAZON-WILDS dataset (Koh et al., 2021) was adapted from the Amazon reviews dataset (Ni et al., 2019), which is a collection of product reviews written by reviewers. While Amazon reviews are always labeled by the star ratings in practice, unlabeled data is a common source of leverage more generally for sentiment classification, with prior work in domain adaptation (Blitzer & Pereira, 2007;Glorot et al., 2011) and semi-supervised learning (Dasgupta & Ng, 2009;Li et al., 2011). In this work, we augment the AMAZON-WILDS dataset with unlabeled reviews, whose star ratings have been removed.

Problem setting. The task is sentiment classification, and we consider generalizing from a set of reviewers to new reviewers at test time. The input x corresponds to a review text, the label y is the star rating from 1 to 5, and the domain d identifies which user wrote the review. For each review, additional metadata (product ID, product category, review time, and summary) are also available. Because the goal is to train a model that performs well across a wide range of reviewers, models are evaluated by their tail performance, concretely, their accuracy on the user at the 10th percentile.

Data. All of the labeled and unlabeled data are taken from the Amazon reviews dataset (Ni et al., 2019). We provide unlabeled data from same domains as the labeled AMAZON-WILDS dataset.

Additionally, we provide unlabeled data from extra reviewers not in the labeled AMAZON-WILDS dataset (extra domains). The domains are split as follows:

1. Source: 1,252 reviewers.

1,334 reviewers.

3. Target (OOD): 1,334 reviewers.

4. Extra (OOD): 21,694 reviewers.

The reviewers in each split are distinct, and all reviewers have at least 75 reviews. The distributions of reviewers in each split are identical. AMAZON-WILDS also includes Validation (ID) and Target (ID) sets which contain data from the source reviewers.

The AMAZON-WILDS dataset has a total of 539,502 labeled reviews across these splits, and we augment the dataset with a total of 3,462,668 unlabeled reviews. For each split of the unlabeled data, we include all available reviews that are written by the reviewer. For the Extra (OOD) split, we include all reviewers with at least 75 reviews that are not in Source, Validation (OOD), or Target (OOD) splits. To filter and process reviews, we followed the same data processing steps as for the labeled data in AMAZON-WILDS (Koh et al., 2021).

Broader context. We focused on providing unlabeled data from OOD domains, including both test and extra domains. Unlabeled data from the test reviewers can be used to develop and evaluate methods for domain adaptation (Ren et al., 2018;Zhang et al., 2019a;Koohbanani et al., 2021), which has been well-studied in the context of sentiment classification (Blitzer & Pereira, 2007;Glorot et al., 2011). While there is limited prior work on leveraging unlabeled data from extra domains, some domain adaptation techniques can be readily adapted to leverage such unlabeled data (Ganin et al., 2016). Finally, we focus on unlabeled data specific to the task in this work, varying only the domains, and this contrasts with the type of unlabeled data used for pre-training in NLP, which is much larger and more diverse (Devlin et al., 2019;Brown et al., 2020).

As a baseline, we consider Empirical Risk Minimization (ERM). ERM ignores unlabeled data and minimizes the average labeled loss. We additionally evaluate ERM with strong data augmentation on applicable datasets, i.e., on IWILDCAM2020-WILDS, CAMELYON17-WILDS, POVERTYMAP-WILDS, and FMOW-WILDS (see Appendix C). ERM with strong data augmentation learns a model h that minimizes the labeled training loss

where A strong is a stochastic data augmentation operation, and measures the prediction loss. We use L L throughout this appendix to refer to the above labeled loss with strong augmentations (on applicable datasets).

For all dataset except CIVILCOMMENTS-WILDS, we sample labeled batches uniformly at random.

In our experiments, we account for class imbalance in CIVILCOMMENTS-WILDS by explicitly sampling class-balanced batches of labeled data when computing L L (h).

Domain-invariant methods seek to learn feature representations that are invariant across domains. These methods are motivated by earlier theoretical results showing that the gap between in-and outof-distribution performance depends on some measure of divergence between the source and target distributions (Ben- David et al., 2010). To minimize this divergence, the methods described below penalize divergence between feature representations across domains, i.e., they encourage the model to produce feature representations that are similar across domains.

Consider a model h = g • f , where the featurizer f : X → F maps the inputs to some feature space, and the head g : F → Y maps feature representations to prediction targets. Domain-invariant methods seek to constrain f to output similar representations for labeled and unlabeled data.

In this work, we adapt all of our domain-invariant methods to use data augmentations on applicable datasets (see Appendix C), and thus the output of f on the labeled batch is

Similarly, the output of f on an unlabeled batch is

Domain-invariant methods seek to minimize some divergence ξ : F × F → R between the labeled data B L and the unlabeled data B U , where the choice of divergence depends on the specific method. The divergence is expressed as a penalty term:

The final objective is a combination of the labeled loss and penalty loss. The balance between the two losses is controlled by hyperparameter λ, the penalty weight.

In our experiments, we study two classical domain-invariant methods, Correlation Alignment (CORAL) (Sun et al., 2016;Sun & Saenko, 2016) and Domain-Adversarial Neural Networks (DANN) (Ganin et al., 2016). These methods are well-known and established, but their performance can be lower than that of newer domain-invariant methods that employ different penalties to encourage the source and target representations to be similar (Jiang et al., 2020;Zhang et al., 2021).

Examples of these newer methods are Joint Adaptation Networks (JAN) (Long et al., 2017), Conditional Domain Adversarial Networks (CDAN) (Long et al., 2018), Collaborative and Adversarial Networks (CAN) (Zhang et al., 2018), and models with Adaptive Feature Norm (AFN) (Xu et al., 2019), as well as methods that minimize the Maximum Classifier Discrepancy (MCD) (Saito et al., 2018) and the Margin Disparity Discrepancy (MDD) (Zhang et al., 2019b).

All of the above methods were developed for the single-source single-target setting, where the source domain is treated as a single distribution, and likewise for the target domain. As each WILDS 2.0 dataset comprises multiple source domains and multiple target domains, it is likely that methods that can leverage this additional structure could perform better. Examples of these methods include Multi-source Domain Adversarial Networks (MDAN) (Zhao et al., 2018) and Moment Matching for Multi-Source Domain Adaptation (M3SDA) (Peng et al., 2019). The DomainBed (Gulrajani & Lopez-Paz, 2020) and WILDS (Koh et al., 2021) benchmarks also extended single-source algorithms like CORAL and DANN to take advantage of multiple source domains in the domain generalization setting, and similar extensions in the domain adaptation setting could be promising.

Correlation Alignment (CORAL). Algorithm 1 describes CORAL, proposed by Sun et al. (2016); Sun & Saenko (2016). CORAL measures the divergence ξ between batches of feature representations in terms of the deviation between their first and second order statistics. Given a labeled batch and unlabeled batch of features B L ∈ R nL×m , B U ∈ R nU×m , define the feature means as

and covariance matrices as

We then compute the CORAL penalty as

We adapted our implementation from DomainBed (Gulrajani & Lopez-Paz, 2020), as done in WILDS 1.0. We note that these implementations compute the penalty as a sum of deviations in means and covariances, whereas Sun et al. (2016); Sun & Saenko (2016) penalize deviations in covariances only. (Sun et al. (2016) considers features that are normalized to zero mean.) On applicable datasets, we also strongly augmented all labeled and unlabeled examples using A strong , whereas Sun et al. (2016); Sun & Saenko (2016) do not explicitly require data augmentations. We add augmentations to allow for a fairer comparison to other methods which use augmentations.

Note that CORAL has also been adapted by Gulrajani & Lopez-Paz (2020); Koh et al. ( 2021) for domain generalization. In particular, where the original CORAL paper defines L penalty as the divergence between just two kinds of batches (labeled and unlabeled), these works define L penalty as the divergence between many kinds of batches, where batches are grouped based on domain annotation d (i) . For simplicity, we followed the original CORAL formulation and differentiate only between labeled and unlabeled batches. We leave leveraging the domain adaptations d to future work.

APPLICABLE DATASETS. We run CORAL on all datasets except GLOBALWHEAT-WILDS and CIVILCOMMENTS-WILDS. We do not evaluate domain invariant methods on CIVILCOMMENTS-WILDS, since the labeled and unlabeled data are drawn from the same distribution. We do not evaluate CORAL on GLOBALWHEAT-WILDS because CORAL does not port straightforwardly to detection settings.

DANN. Algorithm 2 describes DANN, proposed by Ganin et al. (2016). DANN measures the divergence ξ between batches of feature representations using the performance of a discriminator network h d that aims to discriminate between domains. Given a batch of features (either B L or B U ), Finally, Pseudo-Label increases the balance λ(t) between labeled and unlabeled losses over time, initially placing 0 weight on L U (h) and then linearly stepping the unlabeled loss weight until it reaches the full value of hyperparameter λ at some threshold step. We fix the step at which λ(t) reaches its maximum value (λ) to be 40% of the total number of training steps, matching the implementation of Sohn et al. (2020). This scheduling allows the algorithm to initially prioritize the labeled loss, as generated pseudolabels are mostly incorrect while the model has low accuracy. Formally, at step t and given total number of steps T ,

We add augmentations to Pseudo-Label in order to allow for a fairer comparison to other methods that use augmentations. On applicable datasets, we have strongly augmented all labeled and unlabeled examples using A strong , whereas Lee (2013) do not use any data augmentations, i.e., all instances of A strong are replaced with the identity function.

Algorithm 3: Pseudo-Label

L , d

We evaluate Pseudo-Label on all datasets except POVERTYMAP-WILDS, as POVERTYMAP-WILDS is a regression dataset, and hard pseudolabeling does not port straightforwardly to regression tasks.

FixMatch. Algorithm 4 describes FixMatch, proposed by Sohn et al. (2020). Like Pseudo-Label, this algorithm dynamically generates pseudolabels and updates each batch. FixMatch additionally employs consistency regularization on the unlabeled data. While pseudolabels are generated on a weakly augmented view of the unlabeled examples, the loss is computed with respect to predictions on a strongly augmented view. This encourages models to make predictions on a strongly augmented example consistent with its prediction on the same example when weakly augmented. For details about the strong versus weak augmentations we use, see Appendix C.

Formally, the pseudolabel-generating function ψ is given by

Like Pseudo-Label, FixMatch uses confidence thresholding, and unlabeled examples on which the model has low confidence have zero loss. Following Sohn et al. (2020), we keep the balance between labeled and unlabeled losses constant at λ(t) = λ. FixMatch's original authors justify keeping λ(t) at a fixed magnitude (as opposed to slowly increasing λ(t) as in Pseudo-Label) by noting that most predictions made by FixMatch are initially low confidence, so for sufficiently high confidence threshold τ , most unlabeled examples have loss zero, keeping the magnitude of L U (h) initially small. This magnitude grows over time, providing a natural curriculum (Sohn et al., 2020).

We endeavored to match our implementation of FixMatch to the formulation of Sohn et al. (2020)

L , d

We evaluate FixMatch on image classification datasets IWILDCAM2020-WILDS, CAMELYON17-WILDS, POVERTYMAP-WILDS, and FMOW-WILDS.

We do not evaluate FixMatch on other datasets because FixMatch relies on enforcing consistency across data augmentations, which we only define for image datasets (see Appendix C).

Noisy Student. Algorithm 5 describes Noisy Student, proposed by Xie et al. (2020). Unlike Pseudo-Label and FixMatch, which update the model and re-generate new pseudolabels each batch, Noisy Student generates pseudolabels, fixes them, and then trains the model until convergence before generating new pseudolabels. First, an initial teacher model is trained on the labeled data; next, the teacher model pseudolabels the unlabeled data, and a student model is trained on the labeled and pseudolabeled data; finally, the student model becomes the new teacher, and the cycle repeats (see Algorithm 5). Each (teacher, student) pair is termed an iteration; we study the results of two iterations.

We train Noisy Student using hard pseudolabels, which the teacher generates over weakly augmented inputs:

While the teacher generates pseudolabels on a weakly augmented data, the student must make both labeled and unlabeled predictions on noisy (i.e., strongly augmented) data. Following Xie et al.

(2020), we add a dropout layer (p = 0.5) before the student's last layer, randomly corrupting final feature maps. Students thus are trained to be consistent across both data-based and model-based noise. We denote the model with inserted dropout as Dropout • f . Xie et al. ( 2020) add even more model-based noise using stochastic depth; for simplicity, we do not use stochastic depth in our implementation.

We follow the original paper and fix the balance between labeled and unlabeled losses as λ(t) = 1. Noisy Student does not use confidence thresholding.

Note that Xie et al. (2020) use both dropout and strong data augmentations when training the initial teacher on labeled data. We reuse models from our ERM + Data Augmentation experiments as initial teacher models; thus we differ from Xie et al. (2020) in that our initial teachers were trained with strong augmentations, but not dropout (see Algorithm 5).

APPLICABLE DATASETS. We evaluate Noisy Student on all datasets except text datasets CIVILCOMMENTS-WILDS and AMAZON-WILDS.

For GLOBALWHEAT-WILDS and OGB-MOLPCBA, we run Noisy Student without noise from data augmentations.

We report both in-distribution and out-of-distribution performance metrics on all datasets, with the exception of OGB-MOLPCBA, which does not have a separate in-distribution test set. Using the terminology in WILDS (Koh et al., 2021), we consider the train-to-train in-distribution comparison on IWILDCAM2020-WILDS, CAMELYON17-WILDS, FMOW-WILDS, and POVERTYMAP-WILDS, and the average comparison on CIVILCOMMENTS-WILDS and AMAZON-WILDS.

For all experiments, we use the same models for each dataset as in WILDS 1.0:

• IWILDCAM2020-WILDS: ResNet-50 (He et al., 2016).

• CAMELYON17-WILDS: DenseNet-121 (Huang et al., 2017).

• FMOW-WILDS: DenseNet-121 (Huang et al., 2017).

• POVERTYMAP-WILDS: Multi-spectral ResNet-18 (Yeh et al., 2020).

• GLOBALWHEAT-WILDS: Faster-RCNN (Ren et al., 2015).

• OGB-MOLPCBA: Graph Isomorphism Network (Xu et al., 2018).

• CIVILCOMMENTS-WILDS: DistilBERT (Sanh et al., 2019).

• AMAZON-WILDS: DistilBERT (Sanh et al., 2019).

The models for IWILDCAM2020-WILDS, FMOW-WILDS, and GLOBALWHEAT-WILDS were initialized with weights pre-trained on ImageNet. Note that models for CAMELYON17-WILDS were not initialized with ImageNet weights. The DistilBERT models were also initialized with pre-trained weights from the Transformers library.

For each dataset, we set the total batch size (where a batch contains both labeled and unlabeled data) to the maximum that can fit on 12GB of GPU memory (Table 13). For all the methods that leverage unlabeled data, except the pre-training algorithms, we run with 4 steps of gradient accumulation, resulting in a 4× larger effective batch size. For SwAV pre-training, we run with 4 GPUs in parallel, which achieves a similar effect. For masked LM pre-training, we run with the default setting of 256 steps of gradient accumulation. These larger batch sizes deviate from the defaults used in the WILDS paper (Koh et al., 2021). We use these larger batch sizes because methods that leverage unlabeled data tend to use larger batch sizes (Sohn et al., 2020;Xie et al., 2020;Caron et al., 2020) Table 13: The batch sizes of each dataset from the original WILDS 1.0 paper and the batch sizes used in WILDS 2.0, which correspond to the maximum that can fit into 12GB of GPU memory.

For models that use batch normalization, the composition of each batch affects the way in which batch normalization is applied. For CORAL, DANN, and Pseudo-Label, we concatenate the labeled and unlabeled data together in each batch, so the labeled and unlabeled data are jointly normalized.

Dataset \ # epochs ERM 3:1 ratio 7:1 ratio 15:1 ratio Table 14: The number of epochs (complete passes over the labeled data) used for each dataset, specified for the ERM baseline as well as different ratios of unlabeled to labeled data within a batch.

For FixMatch, we jointly normalize the labeled data and the strongly augmented unlabeled data, but we separately normalize the weakly augmented unlabeled data in a separate forward pass; we did two forward passes to keep the overall batch sizes consistent with the other algorithms, as in Table 13, while still fitting in GPU memory. For Noisy Student, MLM pre-training, and SwAV pretraining, the unlabeled data is processed separately from the labeled data, so each batch of labeled or unlabeled data is separately normalized.

We tune each algorithm separately for each dataset by randomly sampling 10 different hyperparameter configurations within the ranges defined below. We early stop and select the best hyperparameters based on the OOD validation performance, which is computed on the labeled Validation (OOD) data for each dataset; we do not use the labeled Validation (ID) data in our experiments. We then run replicates using the best hyperparameters. For computational reasons, we do not tune hyperparameters for the pre-training algorithms, though we tune the finetuning of their resulting pre-trained models as usual.

Learning rates. For all the datasets, except for OGB-MOLPCBA, we multiply the learning rates used in WILDS by the ratio of the effective batch size to the original batch size used in WILDS 1.0. We center the learning rate grid around this modified learning rate r, and search over r • 10 U (-1,1) , where U is the uniform distribution. For OGB-MOLPCBA, we pick r by multiplying the original learning rate by a factor of 10 instead of 4096/32 = 128 (for ERM, which does not have grandient accumulation), because we found that the latter led to unstable optimization.

L 2 -regularization. Across all datasets and methods, we used the same L 2 -regularization strengths used in WILDS 1.0.

Ratio of unlabeled to labeled data in a batch. For all the domain-invariant and self-training methods, we search over the ratio of unlabeled to labeled data in a batch, using the values {3 : 1, 7 : 1, 15 : 1}.

We defined an epoch as a complete pass over the labeled data. This means that the number of batches / gradient steps taken per epoch varies with the ratio of unlabeled to labeled data in a batch, as a higher ratio means that each batch contains fewer labeled examples.

We adjusted the number of epochs accordingly so that the total amount of compute was similar regardless of the ratio of unlabeled to labeled data in a batch. We allocated roughly twice as much compute (i.e., processing twice as many batches) to methods that used unlabeled data, compared to the purely-supervised ERM baseline. Overall, we set the number of epochs based on the WILDS 1.0 defaults, with some upwards adjustments (due to the different batch sizes and the use of unlabeled data) if we found that the best hyperparameter configuration had not converged on the validation set. For DANN, Pseudo-Label, and FixMatch, we compared our results against the results reported in Zhang et al. (2021). Performance was similar for DANN (ours, 39.4%; theirs, 40.0%). For Fix-Match, our implementation performs better (ours, 50.2%; theirs, 45.3%); this is partially due to our use of strong instead of weak augmentation for the labeled data, which increases performance by 0.9%. For Pseudo-Label, our implementation performs worse (ours, 36.1%; theirs, 40.6%), and we believe it is due to variation in hyperparameter tuning.

For Noisy Student, Berthelot et al. (2021) reported significantly lower numbers (ours, 39.7%; theirs, 32.6%). However, this is expected as they trained their models from scratch, whereas we used ImageNet-pretrained models.

We were unable to find comparable results in prior work for CORAL and SwAV pretraining on the real → sketch split. Prior work has shown that these methods can improve performance on other unsupervised adaptation datasets (Sun & Saenko, 2016;Shen et al., 2021). On our DomainNet experiments, we found that SwAV pre-training did improve performance over ERM, though CORAL did not (Table 15).

The self-training methods we evaluate in Section 5 generate a pseudolabel ỹU for each unlabeled example x U and then train on (x U , ỹU ) as if the pseudolabels were true labels. However, these pseudolabels may not be accurate. In this section, we describe how we ran fully-labeled ERM experiments using ground truth labels on the "unlabeled" data to establish informal upper bounds on how well we might expect a standard self-training approach to perform with perfect pseudolabel accuracy.

For four of our datasets (AMAZON-WILDS, CIVILCOMMENTS-WILDS, IWILDCAM2020-WILDS, and FMOW-WILDS), we curated the "unlabeled" data by taking labeled examples and discarding the ground truth labels. For example, all 268,761 of the unlabeled target reviews in AMAZON-WILDS actually have associated star ratings; these are available in our data loaders, but in our main experiments we treat these reviews as unlabeled by not loading the star ratings. We evaluated models trained via empirical risk minimization (ERM) on the combination of the standard labeled training set and the unlabeled data with these hidden labels revealed. For example, in AMAZON-WILDS, we pool together the labeled source examples as well as the unlabeled target examples with ground truth labels, and evaluate ERM models trained on all of that data. As with all of the other experiments in this paper, we evaluate test performance for all datasets on the labeled target splits, so at no point are we training on our actual test examples.

Pooling labeled and unlabeled data. For all datasets, we pooled labeled source examples with examples from the same "unlabeled" split as in our main experiments (Table 2). We computed gradients for labeled minibatches and unlabeled minibatches separately, which means that for models using batch normalization, the labeled and unlabeled data were normalized separately. However, we fixed the labeled to "unlabeled" batch size ratios to match the overall labeled to unlabeled dataset size ratio, so other than the batch normalization effects, the training procedure can be viewed as running ERM on the pooled labeled and "unlabeled" data.

With the exception of IWILDCAM2020-WILDS, detailed below, we followed the procedure in Appendix D.4 to adjust the number of epochs based on the labeled to unlabeled batch size ratios. This resulted in a similar amount of computation allocated to these fully-labeled ERM experiments as the other experiments in Table 2.

Other details. Other experimental details were kept similar to the other experiments in the paper. Specifically, we tuned each experiment by randomly sampling 10 different hyperparameters within the ranges defined in Appendix D.4; the only hyperparameter we tuned in these experiments was the learning rate. We early stopped and selected the best hyperparameters based on the OOD validation performance, and then ran replicates using the best hyperparameters. We also used data augmentation for IWILDCAM2020-WILDS and FMOW-WILDS but not for AMAZON-WILDS and CIVILCOMMENTS-WILDS. (2,927,841 examples). However, this did not improve performance. Using the unlabeled target data, we obtained an average accuracy of 73.6 (± 0.1) and a 10th percentile accuracy of 56.4 (± 0.8), whereas using the unlabeled extra data, we obtained an average accuracy of 73.1 (± 0.1) and a 10th percentile accuracy of 54.7 (± 0.0).

We used the unlabeled extra split (1,551,515 examples). As in our other experiments on CIVILCOMMENTS-WILDS, we accounted for label imbalance by sampling class-balanced labeled and "unlabeled" batches during training. examples. We found that we required twice as many epochs compared to the other unlabeled methods for the fully-labeled ERM training to converge, so we doubled the amount of compute allocated to the fully-labeled IWILDCAM2020-WILDS experiments.

FMOW-WILDS. We used the unlabeled target split (173,208 examples).