Modern machine learning has achieved great progress in various applications, such as computer visual [18,34,66,67,83,91], and natural language processing [17,23,35,61,70,80]. Despite the immense success, existing approaches typically assume that training and testing data are independently sampled from the identical distribution. However, in real-world scenarios, this assumption rarely holds. In image recognition, for example, distribution shifts w.r.t. geographic locations [7] and image background [26] frequently occur and impede models' generalization performance.
Accordingly, domain generalization (DG) [31] is studied to enhance the transferability of deep learning models. A natural idea for DG is to learn invariant representations for same-class samples across a variety of seen domains so as to benefit the classification of unobserved testing domain samples. As a powerful representation learning technique, contrastive learning (CL) [14] aims to obtain class-separated representations and has the potential for DG [41]. In this paper, however, we have observed the limitation of the widely deployed self-contrastive learning (SCL), which aligns the augmentation of the same input. Although SCL has demonstrated success in unsupervised pre-training tasks [14,30,33], it does not naturally fit the domain generalization setting: SCL implicitly assumes the capability to sample instances from the whole data distribution, which does not fit the practical domain generalization scenario where models are fine-tuned using data from specific partial domains. Consequently, SCL struggles to acquire generalizable representations in this context.
To bridge this gap, we propose domain-connecting contrastive learning (DCCL) to pursue transferable representations in DG, whose core insight comes from a recent novel understanding attributing the success of CL to the intra-class representation connectivity [78]. Specifically, we first suggest two direct approaches to improve intraclass connectivity (to be fully explained at the beginning of Section 2) within CL models: applying more aggressive data augmentation and expanding the scope of positive samples from self-augmented outputs to the augmentation of same-class samples across domains. The aforementioned approaches aid in establishing connections among existing domains.
The module above focuses on enhancing intra-class connectivity from the data perspective. However, the embeddings of the unseen
(a) Pre-trained. (b) ERM. (c) SCL. (d) DCCL. testing domains and the ones of the training domains in the same class may still be separated. To address this issue, we make and utilize an observation that the pre-trained models from the large database, unlike the learned maps of Empirical Risk Minimization (ERM), indeed possess the desired intra-class connectivity: the intra-class samples of the training domains and the testing domains are scattered but well-connected, as demonstrated in Figure 1a and Section 3.4. This encouraging observation motivates us to anchor learned maps to the pre-trained model by broadening the augmentation strategies in CL. Furthermore, to close the gap in the representations of pretrained and fine-tuned models, we propose to complement contrastive learning with the generative transformation loss for enriched supervised signals. As a visual illustration, Figure 1 demonstrates the embeddings learned by regular ERM and by the proposed DCCL. ERM embeds the data in a more scattered distribution, and many samples in the central region cannot be distinguished; on the other hand, DCCL well clusters inter-class samples regardless of the domains. It verifies the effectiveness of our proposed DCCL on connecting domains. Our contributions are summarized as follows:
• We analyze the failure of self-contrastive learning on DG and propose two effective strategies to improve intra-class connectivity within CL models. • We propose to anchor learned maps to pre-trained models that possess the desired connectivity of training and testing domains. We further propose generative transformation loss to complement the alignment between learned maps and pre-trained models.
• We conduct extensive experiments on five real-world DG benchmarks with various settings, demonstrating the effectiveness and rationality of DCCL.
The rest of the paper is organized as follows. We introduce the problem formulation and preliminaries in Section 2, present our proposed DCCL in Section 3, show the experimental results in Section 4, discuss the related work in Section 5, and conclude in Section 6.
We first illustrate the core concept of the paper, intra-class connectivity. It refers to the intra-class data connectivity across different domains and resembles the connectivity in CL theory [78], which depicts the preference that samples should not be isolated from other intra-class data of the same classfoot_0 . In the remainder of this section, we introduce the problem formulation and necessary preliminaries for contrastive learning. A thorough review of related work on domain generalization and contrastive learning are deferred to Section 5.
Given 𝑁 observations (from 𝑀 domains), X = {𝑥 1 , . . . , 𝑥 𝑁 } ⊆ X is the collection of input designs, Y = {𝑦 1 , ..., 𝑦 𝑁 } ⊆ Y represents the prediction targets, and the whole dataset 𝐷 𝑠 is denoted as
, where 𝑁 𝑚 is the number of samples (naturally, 𝑀 𝑚=1 𝑁 𝑚 = 𝑁 ) in domain 𝑑 𝑚 and 𝑥 𝑖 is re-indexed as 𝑥 𝑚 𝑖 .
Contrastive Learning (CL) enforces the closeness of augmentation from the same input, compared to other inputs in the representation space. The main components of CL, as summarized in [14,33], include: (i) data augmentation for contrastive views, (ii) a representation map 𝑓 as the data encoder: X → R 𝑑 , (iii) projection head ℎ(•) for expressive representation, and (iv) the contrastive loss for optimization. Given an instance from X, we draw a positive pair 𝑥, 𝑥 + by applying a random data augmentation 𝑎 ∼ A, where A is the pre-specified distribution of random data augmentation maps. As a contrastive concept to positive samples, a negative pool N 𝑥 is the set of augmented samples randomly drawn from the whole dataset X. To ease the construction of the CL loss, we denote 𝑝 (𝑥) as the distribution of 𝑥, 𝑝 𝑥, 𝑥 + as the corresponding joint distribution of the positive pairs, and 𝑝 𝑛 (𝑥 - 𝑖 ) ("n" is shorthand for "negative") as the distribution for the negative sample 𝑥 - 𝑖 ∈ N 𝑥 , which are all independent and identically distributed (i.i.d.). Let 𝑧 denote The most common form of the CL loss (L CL ) adapts the earlier InfoNCE loss [58], formulated as:
where 𝜏 > 0 is the temperature parameter. The CL loss is typically used in the unsupervised [14,30,33] or supervised [40] pretraining setting. To adapt it to domain generalization [13,41,88], the full model is also required to learn from supervised signals.
Thus, it is intuitive to combine the CL loss with the empirical risk minimization (ERM) loss L ERM as the following objective:
where 𝜆 is the regularization hyper-parameter during training. In practice, L ERM is usually chosen as the softmax cross entropy loss to classify the output embedding 𝑧. We follow the classical setting [41] in this paper, which includes both classification loss and selfsupervised regularization loss.
In this section, we present the details of DCCL, which learns robust representations for tackling distribution shifts across domains. We first comment on the failure of directly applying self-contrastive learning to DG in Section 3.1. Followed by the implications from learning theory in Section 3.2, we propose two complementary strategies to improve intra-class data connectivity in Section 3.3 to initialize our domain-connecting CL. Then in Section 3.4, we introduce pre-trained model anchoring to further utilize the intra-class connectivity of the representation output by the pre-trained model. A generative transformation module is designed to assist the anchoring and help encode the essential information in the pre-trained representation. The overall framework of DCCL is shown in Figure 2, which integrates data and model information for generalization.
Learning in Domain Generalization
Self-contrastive learning, which aligns the augmentation views of the same input, has achieved impressive performance in unsupervised pre-training tasks [14,30,33]. However, it does not naturally fit the domain generalization setting since it assumes the ability to sample 𝑥 from the whole data distribution: in the training stage of domain generalization, we instead are only able to access partial domains. This mismatch can lead to suboptimal performance in DG if the users mechanically adopt the classical CL loss.
We provide a linearly separable toy example in Figure 3 to show the deficiency of SCL. In particular, even attaining the optimal CL loss (1) cannot guarantee good DG performance, where only partial domains are involved in training. We detail the coined data distribution as follows.
Example 3.1 (SCL does not help domain generalization.). Let the label collection Y be {-1, 1} and the portions of two classes be both 0.5. Assume there are two domains 𝑑 1 and 𝑑 2 : if a sample 𝑋 = (𝑋 1 , 𝑋 2 ) ∈ R 2 with label 𝑌 is from domain 𝑑 1 , its conditional distribution will be specified as
In domain 𝑑 2 the distribution of 𝑋 1 , 𝑋 2 can be analogously represented. Considering only domain 𝑑 1 is involved in training, we construct a map 𝜑 (𝜃 (𝑥)) := (cos (𝜃 ) , sin (𝜃 )) with 𝜃 (𝑥) = (𝑥 1sgn(𝑦)) 𝜋 for the weak augmentation setting and 𝜃 (𝑥) = (sgn(𝑥 1 ) + 𝑦) 𝜋 3 for the aggressive augmentation setting. The map 𝑓 ℎ = 𝜑 • 𝜃 attains perfect alignment of intra-class samples and maximal uniformity (representations of the augmented samples are uniformly distributed on the corresponding circle arcs) on the 1-sphere S 1 := 𝑥 ∈ R 2 : ∥𝑥 ∥ 2 = 1 . Based on the derivation in [76], 𝑓 ℎ will minimize the CL loss (1). Figure 3 illustrates the example, where slashes and spots are used to represent domains 𝑑 1 and 𝑑 2 ; orange and blue rectangles respectively denote classes 1 and -1. For ease of analysis, we specifically consider the case that only domain 𝑑 1 is involved in training. Note that adding more domains does not affect the conclusion of our analysis. In Figure 3a, We can observe that when applying weak augmentation, the new representations for domain 𝑑 2 do not reflect the class information and even have the opposite signs as domain 𝑑 1 . On the other hand, in Figure 3b, with aggressive augmentation, the intra-class samples of different domains are connected. In this case, the optimal representations learned on domain 𝑑 1 can also reflect the accurate class information of testing domain 𝑑 2 .
We can conclude that the usage of classical SCL with weak augmentation does not necessarily lead to good DG performance; empirical verification is provided in Section 4.4 as well. A similar limitation is observed in invariance-based DG methods [63]. The key to the problem lies in improving the intra-class connectivity (achieved by aggressive augmentation in this example) across domains.
(a) Weak Augmentation. (b) Aggressive Augmentation.
Building on these observations, we delve deeper into understanding these limitations. Specifically, we demonstrate that intra-class connectivity is crucial for reducing the intra-class representation variance Var(𝑓 ℎ (𝑥)|𝑦), as outlined in Proposition C.1 and further analyzed in Appendix C due to space limitations. Reducing this variance enhances domain generalization by promoting stable feature representations that are less influenced by domain-specific variations. This theoretical framework of connectivity motivates us to re-examine the failures of SCL discussed in the previous subsection, focusing on the connectivity perspective to uncover potential solutions and improvements.
With regard to domain generalization, if all intra-class samples can be clustered together across domains and the intra-class variance shrinks to zero in CL, we then automatically obtain the generalizable representations. We observe that SCL in the previous example fails to obtain intra-class connectivity due to insufficient data augmentation and domain-separated (rather than classseparated) representations, which ultimately cause poor generalization performance. We thus propose two approaches to improve intra-class connectivity: (i) applying more aggressive data augmentation and (ii) expanding the scope of positive samples, from solely self-augmented output 𝑎(𝑥) to the augmentation of intra-class samples across domains. In applying CL, proper data augmentation can help "connect" two different samples 𝑥 𝑖 , 𝑥 𝑗 within the same class, which technically means there exists a pair of augmentation maps 𝑎 𝑖 , 𝑎 𝑗 so that 𝑎 𝑖 (𝑥 𝑖 ), 𝑎 𝑗 (𝑥 𝑗 ) are close to each other. Consequently, in optimizing the CL loss (1) the representations 𝑓 ℎ (𝑥 𝑖 ), 𝑓 ℎ (𝑥 𝑗 ) will be pushed close since
In other words, as a ladder, 𝑎 𝑖 (𝑥 𝑖 ) and 𝑎 𝑗 (𝑥 𝑗 ) connect the two samples 𝑥 𝑖 , 𝑥 𝑗 , and analogously all the samples within the same class can be connected by proper data augmentation. Similarly, expanding the scope of positive samples can help connect the samples from different domains but same classes, and thus enhance the intra-class connectivity. CL later on pushes their learned representations to cluster thanks to the CL loss.
We remark IRM [1] proposed a similar idea of leveraging the intra-class sample similarities, while the CL theory removes the assumption in IRM that the marginal distribution of sample 𝑥 on source domains should be the same on target domains, and thus is theoretically more applicable to DG.
Inspired by the analysis above, we propose two direct approaches to improve intra-class connectivity: (i) applying more aggressive data augmentation and (ii) expanding the scope of positive samples, from solely self-augmented output 𝑎(𝑥) to the augmentation of intra-class samples across domains.
For the first approach, despite the fact that data augmentation in DG (e.g., horizontal flipping) has already been a standard regularization technique [10,31,73], the choice of data augmentation, we emphasize, matters for CL in the DG setting. We naturally need a larger augmentation distribution A to connect 𝑎 𝑖 (𝑥 𝑖 ) and 𝑎 𝑗 (𝑥 𝑗 ) since 𝑥 𝑖 , 𝑥 𝑗 can be drawn from different domains. The effect of data augmentation intensity is evaluated through the ablation studies in Section 4.3.
Motivated by supervised CL [20,32,40], we further introduce cross-domain positive pairs into CL to bridge the intra-class samples scattered in different domains. Specifically, we not only consider the correlated views of the same data sample as positive pairs but also the augmented instances from other intra-class samples across domains. The positive sample 𝑥 + will now be conditionally independent of 𝑥, and the positive pairs have the same conditional distribution 𝑝 (1) (𝑥 + |𝑦) = 𝑝 (𝑥 |𝑦) 3 (the specific distribution of the positive sample 𝑥 + in this subsection will be denoted with a superscript (1)); in other words, 𝑥 + can now be the augmentation view of a random sample within the same class 𝑦 of 𝑥. With the joint distribution of 𝑥, 𝑥 + denoted as 𝑝 (1) 𝑥, 𝑥 + = ∫ 𝑦 𝑝 (1) (𝑥 + |𝑦)𝑝 (𝑥 |𝑦)𝑝 (𝑦)d𝑦, the primal domain-connecting contrastive learning (DCCL) objective L (0)
DCCL can be formulated as:
Unlike supervised CL, which forms positive pairs from different views within the same domain, our method incorporates intraclass samples across domains, effectively improving intra-class connectivity from a data perspective. The term,log exp 𝑧 • 𝑧 + /𝜏 , corresponding to alignment in loss (3), can now push the intra-class samples from different domains together.
Up to now, we have not addressed the core challenge in DGlack of access to the testing domains in training: CL is originally designed for the self-supervised scenario where a huge amount and wide range of data is fed to the models. However, in the context of domain generalization, the model is just fine-tuned on limited data within partial domains. Consequently, the mechanism of CL can only contribute to the clustering of representations in the seen domains, while the embeddings of the unseen testing domains and the ones of the training domains in the same class may still be separated.
Interestingly, the intra-class connectivity for representations, the desired property in CL, seems to exist at the beginning of the fine-tuning. We observe the phenomenon when visualizing the representations obtained from the pre-trained model using t-SNE [68] in Figure 1a, which thereby motivates our design in this subsection. We find that mapped by the initial pre-trained model ResNet-50, intra-class samples of the training domains and the testing domains are scattered while well-connected.
We attribute the phenomenon to the effective representations returned by pre-trained models, which reasonably model the pairwise interactions among samples and thus draw target domains closer to source domains. To verify the effectiveness of the representations, we design a quantitative metric to evaluate whether the pre-trained space is "well-connected", by turning to the concept of "connectivity" in graphs. Details can be found in Appendix A. 4.
As for the model design, the phenomenon motivates us to better utilize the pre-trained model 𝑓 pre for stronger intra-class connectivity in the mapped representations obtained from 𝑓 . We propose to make use of pre-trained models as data augmentation in a disguised form: data augmentation works on the raw data while we can further "augment" the representation 𝑥 via the model 𝑓 pre .
In mathematical language, in additional to the augmented sample 𝑥 + defined in the last subsection, we further incorporate the pretrained embedding 𝑧 pre = ℎ • 𝑓 pre (𝑥) into the definition of feasible positive embeddings 𝑧 (2),+ , which expands the scope of the previous positive embeddings 𝑧 + (the superscript (2) implies the different distribution compared to 𝑧 + in the last subsection). In particular, for a given 𝑥, we decide the form of the newly coined positive embedding 𝑧 (2),+ as:
With the distribution of the extended positive embedding denoted as 𝑝 (2) 𝑧 (2),+ (the positive pairs 𝑥, 𝑥 + still follow 𝑝 (1) (𝑥, 𝑥 + )), the proposed DCCL loss L DCCL can be written as:
where 𝑝 (2) 𝑧, 𝑧 (2),+ is the joint distribution of 𝑧, 𝑧 (2),+ constructed in this subsection. Our proposed L DCCL manages to mine the supervised signal at the inter-sample level, where we align the positive pairs (composed of different samples) while pushing apart the samples in a negative pool. Echoing the findings in [88], which point out aligning positive pairs across vastly different domains often results in poor performance, our research similarly identifies a substantial gap in the representations of pre-trained and fine-tuned models. Direct alignment using CL as evidenced by our empirical evaluation, tends to be sub-optimal. In response, we introduce the concept of variational generative transformation loss to comprehend the transformation process and bridge these representational gaps. Additionally, the generative transformation module is designed to reconstruct the features of the pre-trained model at an intra-sample level. This complements the inter-sample level supervision provided by contrastive loss. The module, with its associated loss function, intends to provide a more enriched supervised signal, encapsulating crucial within-sample information. In turn, it serves as a pivotal proxy objective that facilitates model anchoring in Eq. 4.
To simplify the notation of the transformation, we abuse the previous notation {𝑧, 𝑧 pre } for the output embedding from a certain learned/pre-trained model layer, omitting the corresponding layer denotation. 𝑧 pre is the fixed supervised signal provided by the pretrained model.
With the notation {𝑧, 𝑧 pre }, we introduce the following variational generative model to parameterize the map 𝑔 : 𝑧 ↦ → 𝑧 pre relating the representation manifolds formed by (the first several layers of) the learned map 𝑓 and the fixed pre-trained model 𝑓 pre . In particular, 𝑔 is composed of an encoder 𝜙 modeling a tunable conditional distribution 𝑞 𝜙 (𝑧 lat | 𝑧) of 𝑧 lat and a tunable decoder 𝜓 mapping 𝑧 lat back to 𝑧 pre , in which 𝑧 lat ∈ R 𝑑 ′ is the latent representation of the generator. Similar to the training of a regular variational autoencoder (VAE) [43], the latent variable 𝑧 lat will be sampled from 𝑞 𝜙 (𝑧 lat | 𝑧); we can then project 𝑧 lat to the pretrained embedding space via decoder 𝜓 for reconstruction. Our variational generative transformation loss L Gen DCCL is designed as:
where 𝑝 (𝑧 lat ) represents the pre-specified prior distribution of 𝑧 lat , 𝑝 𝜓 𝑧 pre | 𝑧 lat is decided by the "reconstruction loss" ∥𝑧 pre -𝜓 (𝑧 lat ) ∥ 2 , and the KL divergence term corresponds to the variational regularization term to avoid mode collapse. The workflow of our proposed generative transformation is shown in Figure 4. Finally, to benefit the representation learning through both generative transformation and our improved contrastive leaning, we set our ultimate objective as:
where 𝜆 and 𝛽 are coefficients to balance the multi-task loss. The ablation studies in Subsection 4.3 verify the effectiveness of each component.
In this section, we empirically evaluate the performance of our proposed DCCL, intending to answer the following research questions: • RQ1: Does DCCL enable networks to learn transferable representation under distribution shifts? • RQ2: How do the various components and experimental choices within our DCCL influence the performance? • RQ3: How good is the generalizability of our proposed DCCL under different circumstances (e.g., varying label ratios, backbones, modalities)? • RQ4: Does DCCL truly establish connections between cross-domain representations?
We exhaustively evaluate out-of-domain (OOD) accuracy of DCCL on various representative DG benchmarks as in [10,11,13,88]: OfficeHome [71], PACS [45], VLCS [26], TerraIncognita [7], and Do-mainNet [59]. The details of the data sets are shown in Appendix A.1. For fair comparison, we strictly follow the experimental settings in [10,13,31,88] and adopt the widely used leave-one-domain-out evaluation protocol, i.e., one domain is chosen as the held-out testing domain and the rest are regarded as source training domains. The experiment results are all averaged over three repeated runs. Following DomainBed [31], we leave 20% of source domain data for validation and model selection. As in previous works [11,88], we use the ResNet-50 model pre-trained on ImageNet by default, and our code is mainly built upon DomainBed [31] and SWAD [10]. All baselines employ identical pre-trained backbones and dataset splits. We apply the same level of data augmentation across all
We provide comprehensive comparisons with a set of strong baselines on the domain generalization benchmarks PACS and Of-ficeHome, as shown in Tables 1 and 2, with results for TerraIncognita, VLCS, and DomainNet datasets deferred to Appendix A.2 due to space limitations. The methods in each table are ranked based on their performance on the dataset. The baselines cover a broad and comprehensive range, including improved learning policies [2,46], enhanced augmentation methods [86,95], and domain invariant learning [1,22] from both data [88] and model [11] perspectives.
We observe our proposed method achieves the best performance across different kinds of baselines: the metrics are 44.0 (ERM)→47.0 (Best Baseline)→47.5 (Ours) on DomainNet, 77.3→79.6→80.0 on VLCS, and 47.8→52.9→53.7 on TerraIncognita. The results of the intermediate columns in the tables represent performance on the testing domain. For example, "A" in Table 1 denotes testing on domain Art and training on Photo, Cartoon, and Sketch. The final result is averaged over all domains. The symbol + in the tables is used to denote that the reproduced experimental performance is clearly distinct from the reported one (such as "PCL + " in Table 5).
All the baselines are sorted in ascending order of their performance.
We have the following findings from the tables. (i) We find that DCCL substantially outperforms all the baseline methods concerning OOD accuracy. This indicates the capability of DCCL to extract transferable representation for generalization under distribution shift. (ii) We notice most baselines make explicit use of domain supervision, while only a few methods such as RSC [39], SagNet [55], COMEN [13], SWAD [10], MIRO [11] and our DCCL do not. The excellent performance of our DCCL may reveal previous works do not well utilize the domain information and there is still much room for improvement. (iii) We note that PCL [88] (Proxy Contrastive Learning) has utilized the potential of CL, aligns embeddings of different samples into domain centers, and consistently achieves good performance. Meanwhile, MIRO [11] also preserves the pre-trained features by adding the mutual information regularization term and attains satisfactory performance. However, because of their deficiency to connect cross-domain representations, our method manages to improve upon the success the previous baselines had.
In this part, we investigate the effectiveness of the proposed DCCL in Table 3 by evaluating the impact of different components. We denote the Cross-Domain Contrastive learning in Section 3.3 as CDC (with more aggressive data augmentation and cross-domain positive samples), Pre-trained Model Anchoring in Section 3.4 as PMA, and Generative Transformation in Eq. 5 as GT. The ablation results are summarized in Table 3. The check mark in the table indicates the module is incorporated. We note that our improved contrastive learning loss in Eqn. (4) has two components: CDC and PMA. The overall improvement of the loss is substantial: 70.6 → 72.9. From the table, we can observe that all the components are useful: when any one of these components is removed, the performance drops accordingly. For example, removing PMA module leads to significant performance degeneration, which verifies the importance of anchoring learned maps to pre-trained models. We can then find the combination of PMA and GT leads to the highest improvement in the ablation, which indicates GT and PMA modules complement each other in an effective way. The finding is also consistent with our motivation for generative transformation loss. Moreover, we also evaluate self-contrastive learning. The experimental results indicate that self-contrastive learning will distort the learned embeddings and hamper performance. Besides, the experiment without aggressive data augmentation also validates the effectiveness of stronger data augmentations we suggest in Section 3.3. In this paper, we increase the intensity of data augmentation operations beyond what is used in typical supervised learning to achieve more aggressive data augmentation. More details and further experimental verification can be found in Table 13 in the Appendix. The efficiency and impact of hyper-parameters are shown in Appendix A.6 and A.7. We note that our method exhibits similar or even lower time and memory costs while stably outperforming baselines regardless of different hyper-parameters. Additional experimental details and explanations regarding our choices for VAE structures, contrastive learning techniques within DCCL, cross-domain examples in CDC, and the Wilds Benchmark can be found in Appendix A.5. The experimental results further verify the robustness of our proposed DCCL.
Generalization ability (RQ3). To verify the generalizability of our proposed DCCL, we first conduct experiments 4 with different label ratios (the percentage of labeled training data) and backbones. (i) In Table 4, we find DCCL can obtain consistent improvement over baselines, in both cases of 5% and 10% label ratios. Our method yields a 16.3 and 14.6 absolute improvement compared with ERM. We can observe that as the number of available labels reduces, the model benefits more from our DCCL (compared with previous 67.6→73.5 increase under 100% label ratio in Table 2). (ii) In Table 5, we test the performance with a new backbone, ResNet-18 (previously ResNet-50) 5 . We find that even though the baselines' relative ordering changes significantly, our model still performs the best, showcasing the robustness thereof. We further observe replacing the ResNet-18 pre-trained representations to the larger ResNet-50 ones ("mismatch" between the backbone used for fine-tuning and the pre-trained representations) will cause substantial performance drop 67.5 → 62.6. The superior performance of DCCL on more backbones (RegNet, ViT) are shown in Table 10 in Appendix.
Analysis of the representations in DCCL (RQ4). Here we analyze the representations in DCCL to provide more insights. In Figure 1, we utilize t-SNE [68] to visualize the embeddings in the pretrained model, ERM, SCL and our DCCL. We observe that mapped by the original pre-trained model ResNet-50, the intra-class samples of the training domains and the testing domains are scattered while well-connected. However, in the ERM model, many samples in the testing domain are distributed in the central part of the plot, which is separated from the training samples. There is a clear gap between the training and the testing domains. As for SCL, it seems to harm the learned embedding space and distort the class decision boundary. Our proposed DCCL can effectively cluster the intra-class samples across domains. We then visualize the embeddings in ERM, PCL, and our DCCL on the testing domains in Appendix A.3. Our DCCL learns discriminative representations even in the unseen target domain by enhancing intra-class connectivity, which is unaddressed in ERM and PCL.
In this section, we review the related works in domain generalization and contrastive learning.
Improving model robustness under distribution shifts has also been extensively studied in domains such as recommender systems [3,79,81,82,84,90], federated learning [4,5], and graph learning [6,16,51,52,77,87,97]. The goal of DG is to enable models to generalize to unknown target domains under distribution shifts. The related literature can be split into several categories as follows.
(i) The first line of work focuses on learning policies. One strategy is meta learning [27], which adapts to new environments rapidly with limited observations; the meta-optimization idea was thus introduced in DG [2,46,60] to generalize to future testing environments/domains; another widely-studied strategy is ensemble learning [10,19], claiming DG can benefit from several diverse neural networks to obtain more robust representations. (ii) The second line of work is data augmentation. Many fabricated or learnable augmentation strategies [48,72,86,95] were developed to regularize and enhance deep learning models. In our paper, we verify more aggressive augmentation can lead to better representations in CL as well. (iii) The last series of work is domain invariant learning. Researchers seek to learn invariances across multiple observed domains for improved generalization on target domains. The commonly used approaches include domain discrepancy regularization [47,94] and domain adversarial learning [28,50,54]. Recently, MIRO [11] began to explore the retention of pre-trained features by designing the mutual information regularization term. The paper [53] also utilized the concept connectivity to build up the method. However, their concept of "connectivity" based on joint distribution clearly differ from our paper. Therefore the theoretical motivation behind two papers are indeed different. Moreover, the methods proposed are different. Except for the common strategy of strong augmentation recommended by the contrastive learning theory paper [78], our proposed methods are different from the ones in [53]. They propose two nearest-neighbor-based methods for constructing positive pairs, while our main contribution lies in the exploitation of both the pre-trained models and the intra-class data connectivity.
Contrastive learning (CL) [14] aims to learn discriminative sample representation by aligning positive instances and pushing negative ones apart. As a promising self-supervised learning paradigm, CL is widely used in unsupervised pre-training to improve the performance of downstream tasks [9,14,15,29,30,33,36,37,49,85]. SimCLR [14] is the CL framework that first reveals the projection head and data augmentation as the core components to learn invariant representation across views. MoCo [33] proposes to build a dynamic queue dictionary to enlarge batch size for effective learning. There are also works [20,32,40] adapting CL to the supervised setting to leverage label information.
The capability of CL to obtain class-separated representations has also motivated the application in domain generalization. Sel-fReg [41] introduced a new regularization method to build selfsupervised signals with only positive samples; PCL [88] proposed a proxy-based approach to alleviate the positive alignment issue in CL; COMEN [13] used a prototype-based CL component to learn the relationships between various hidden clusters. However, the role of CL in domain generalization is not yet well explored, and our work is dedicated to shedding some light on the understanding of its effect from a intra-class connectivity perspective.
In this paper, we revisit the role of contrastive learning (CL) in domain generalization and identify a key factor: intra-class connectivity. We further realize this characteristic of representations can be attained from two aspects, data and model. On the data side, we analyze the failure of directly applying CL to DG and propose two strategies to improve intra-class connectivity: (i) applying more aggressive data augmentation and (ii) expanding the scope of positive samples. On the model side, to alleviate lack of access to the testing domains in training, we propose to anchor learned maps to pre-trained models which enhances the desired connectivity between training and testing domains. Generative transformation is further introduced to complement the pre-trained alignment. Consequently, we combine the pieces together and propose DCCL to enable robust representations in the out-of-domain scenario. Extensive experiments on five real-world datasets demonstrate the effectiveness of DCCL, which outperforms a bundle of baselines. Here we elaborate the detailed experimental setup of our paper. Following DomainBed [31], we split 80%/20% data from source domains as the training/validation set. The best-performing model on the validation set will be evaluated on the testing target domain to obtain the test performance. The statistics of the experimental datasets are shown in Table 6. We list the number of images, domains, classes in each dataset. The proposed model is optimized using Adam [42] with the learning rate of 5e-5. The hyper-parameter 𝜆 is searched over {0.1, 1, 2, 5}, and 𝛽 is tuned in the range of {0.01, 0.05, 0.1}. The temperature 𝜏 is set to 0.1 by default. For the projection head used for contrastive learning, we use a two-layer MLP with ReLU and BatchNorm. Regarding variational reconstruction, following [11], we employ a simple yet effective architecture, in which the identity function is used as mean encoder and a bias-only network with softplus activation for the variance encoder. More intricate architecture can be explored in the future. Following [31], for all the datasets except DomainNet, we train the model for 5000 steps. For the DomainNet dataset, we train the model for 15000 steps. Other algorithm-agnostic hyper-parameters such as the batch size are all set to be the same as in the standard benchmark DomainBed [31]. For batch construction, we sample the same number of samples from each training domain as in DomainBed [31]. Generative Transformation is done for all 4 layers in ResNet-18/50. The experiments are all conducted on one Tesla V100 32 GB GPU. The baseline results are taken from the original papers. If the results were not available, we reproduced them for fair comparisons. For the data augmentation strategy, previous works usually adopted random cropping, grayscale, horizontal flipping and random color jittering. In this paper, we simply increase the intensity of random color jittering to achieve more aggressive data augmentation on all datasets. Developing stronger and more adaptive augmentation methods for contrastive learning on DG may further enhance the performance.
We put the experimental comparisons with state-of-the-art baselines on TerraIncognita, VLCS, and DomainNet data sets respectively in Tables 7, 8, and 9. The symbol + in the tables is used to denote that the reproduced experimental performance is distinct from the originally reported one such as "PCL + " in Table 9. We can observe our proposed DCCL still surpasses previous methods, which is consistent with the conclusion in the main text and successfully verify the effectiveness of our proposed method.
We demonstrate the embeddings of ERM, PCL, and our DCCL methods on the testing domain in Figure 5. ERM, among the three methods, has the most samples distributed in the central area which cannot be distinguished. For the embedding of contrastive-learning-based baseline PCL, there are fewer samples distributed ambiguously. However, the class clusters are not compact and the class boundaries are not clear. By contrast, our DCCL learns discriminative representations even in the unseen target domain by enhancing intra-class connectivity in CL.
Our motivation to utilize pre-trained models for better connectivity is intuitive: we consider pre-trained model can return effective representations modeling the pairwise interactions among images, which thus draws target domains closer to source domains. To verify the motivation, we conduct experiments to evaluate whether the pre-trained model is "well-connected".
(1) We design a quantitative metric to help evaluate whether the pre-trained space is "well-connected". For images within the same class, we take those images as nodes and construct a graph, only connecting two nodes when their distance on the pre-trained space is smaller than a threshold. We denote the smallest possible threshold which makes the graph connected as 𝜏, and denote the mean and the std of the pairwise distances respectively as 𝜇 and 𝜎. We can thus use (𝜏 -𝜇)/𝜎 as a metric to describe the connectivity of the representations. (2) We report the mean (max) metrics (the smaller, the better) of each class for ERM and pre-trained model on PACS, VLCS, and Terra.; the values for ERM are 1.37 (2.68), 1.78 (2.15), and 3.31 (3.56), for pre-trained model 0.54 (0.81), 0.46 (0.62), and 0.63 (0.76). The results confirm the pre-trained space is well-connected. Furthermore, the variation in performance improvement across different datasets can be attributed to differences in connectivity. We define a measure to evaluate connectivity in Appendix A.4 where lower values indicate better connectivity. For the pre-trained (ERM) model, the connectivity measure we have is 0.54 (1.37) for PACS and 0.49 (2.85) for OfficeHome. A larger discrepancy in connectivity between ERM and the pretraine model (
1.37 0.54 v.s. 2.85 0.49 ) allows for greater potential for improvement. (a) ERM (b) PCL (c) DCCL
Choices of VAE structures. In our experiments, using more advanced VAE structures like HFVAE [25] (72.7) and IntroVAE [38] (73.1) will yield worse results than vanilla VAE (73.5), which may be attributed to the increased training difficulty.
Choices of contrastive learning methods. SimCLR is denoted as "SelfContrast" in Table 4. Our proposed DCCL (73.5) turns out to outperform other representative SSL approaches: SimCLR [14] (68.9 in Tab. 4), MoCo [33] (69.7), BYOL [30] (70.7), SwAV [9] (71.5).
Further justification of cross-domain contrast (CDC). To further justify cross-domain contrast (CDC), we also implement a baseline using within-domain positive samples only, and the accuracy drops remarkably compared to CDC (71.8 → 70.4). In addition, we include an oracle experiment with solely cross-domain positive pairs and observe comparable performance (71.8 → 71.9). It may require careful design to make good use of domain information to obtain improvements.
Choices of pre-trained backbone and resources. In Table 10, we present additional experiments on Instagram (3.6B) pre-trained RegNet [64] and CLIP (400M) pre-trained ViT [24]. Compared to PCL, which ignores the pre-trained information, DCCL achieves consistent and substantial improvement on imagenet pre-trained models. And when applied to Instagram and CLIP, the improvement becomes remarkably larger. These indicate the importance of the pre-trained information, and more abundant the pre-training resources, the stronger the pre-trained information is needed. Further Experiments on the Wilds Benchmark.
We also test the OOD performance of our proposed DCCL using the Camelyon and iWildCam datasets from the Wilds benchmark with the pre-trained ResNet-50 network. In Table 11, DCCL demonstrate a consistent and substantial improvement in performance on the more challenging datasets. Here we additionally performed an ablation study on the VLCS dataset, as shown in Table 12, where the performance gain above SWAD is relatively smaller. These results further confirm that the three components we identified contribute consistently to the effectiveness, as detailed in our paper. In Table 3 of the paper, we've presented an ablation study on aggressive augmentation. Previous works usually adopted random cropping, grayscale, horizontal flipping and random color jittering. In this paper, we simply increase the intensity of random color jittering to achieve more aggressive data augmentation on all datasets. Here, we provide additional validation in Table 13 by showcasing the performance of ERM and our DDCL on the OfficeHome dataset under various augmentation scenarios: without augmentation, with standard augmentation, and with aggressive augmentation. Notably, aggressive augmentation proves advantageous for our DDCL while detrimental to ERM compared to standard augmentation. Stronger and more adaptive augmentation methods for contrastive learning on DG will be explored to further enhance the performance in the future.
The algorithmic complexity of our method and its baselines is complex due to factors like Feature Extraction time and Loss Calculation time. Feature extraction is consistent across all baselines, including ERM, and is a significant part. For the loss calculation, given a batch size of and a hidden dimension, and using contrastive loss calculated over batch pairs, the complexity is 𝑂 (𝐵 2 𝐷), which is uniform across all contrastive learning methods.
An intuitive graph-based measure to assess the intra-class connectivity of a given model is discussed in Appendix A.4
Unlike the classical setting in self-supervised CL, in DG we can access the label 𝑦 in training.
We select a few of the most representative methods as baselines.
For semantic information matching, pre-trained representations in DCCL are generated from the same backbone model used for fine-tuning.
https://github.com/facebookresearch/DomainBed