Pretraining image classification networks on the Imagenet dataset has led to visual representations that transfer to other tasks (Girshick et al., 2014;Long et al., 2015;Vinyals et al., 2015;Antol et al., 2015;Zhu et al., 2016). However, such classification based pretraining requires a large amount of human-annotated data which is hard to obtain at scale. In contrast, captioned image data is an informationdense source of supervision that is relatively cheap to collect and plentiful on the internet. Therefore, recent methods have used joint vision-language pretraining to learn representations from image-caption pairs (Desai and Johnson, 2021;Sariyildiz et al., 2020). However, methods such as VirTex (Desai and Johnson, 2021) which train on complex language modeling tasks such as masked language modeling, token classification, and captioning fail to align features in a common latent space.
matching image-caption pairs. However, contrastive learning in vision-language pretraining still has some limitations as it seems to be most effective only with large scale data, and it requires a large number of negative image-caption pairs during training. Our work aims to address and explore these two limitations by proposing CLIP-Lite, an information efficient variation of CLIP that is useful even in smaller data regimes, does not rely in as many negative sample pairs during training, and provides comparable or superior performance on standard benchmarks against other methods trained at the same scale. Our work is motivated by the observation that multiple contrastive objectives maximize a lower-bound on the mutual information between two or more views of the same datum (Wu et al., 2020). CLIP particularly maximizes the mutual information between the image and its caption by using a mutual information lower bound based on InfoNCE (Oord et al., 2018). The InfoNCE bound has seen wide adoption due to its favorable properties such as stability and low variance. However, the the bound is theoretically loose in cases when the true mutual information is larger than log K where (K -1) is the number of negative samples used for training. The negative pairs can be randomly sampled but usually a large amount of negative pairs are required to have a good estimate of the mutual information between the two input streams, and hence the need for rather large batch sizes (Bachman et al., 2019;Chen et al., 2015) or memory-banks (Chen et al., 2020b;Tian et al., 2019;He et al., 2020).
We instead adopt a lower bound based on Jenssen Shannon Divergence to maximize the mutual information (Hjelm et al., 2018;Nowozin et al., 2016), thus requiring no more than one negative example pair for each positive example pair. This reduces the number of negative examples in a training batch to O(n), where n is the batch size.
In contrast, CLIP uses O(n 2 ) negative example pairs per batch. Figure 2 (right) illustrates this difference. We implement this strategy and demonstrate thoroughly the efficacy of CLIP-Lite through experiments on several tasks and datasets at various scales. Our method demonstrates impressive data efficiency and is able to outperform CLIP trained on the entire COCO-Captions dataset while only training on 20% of the same dataset. We also demonstrate that CLIP-Lite can be used as a good source of pretrained features by showing good generalization on Pascal VOC and Imagenet classification. We also show that the visual feature backbone of CLIP-Lite can be finetuned in the iNaturalist dataset to match top performances on this benchmark with caption supervision pretraining. Furthermore, we show that CLIP-Lite leads to good visual features for image retrieval compared to regular CLIP trained on COCO Captions. We also demonstrate that CLIP-Lite enables the removal of concepts from visual representations which we show can be applied in bias mitigation. Our work extends and complements the work using contrastive learning, especially addressing the computational requirements of the original CLIP model in terms of memory overhead through minimizing the number of negative sample imagetext pairs required during training and shows its effectiveness in smaller data regimes including for zero-shot learning on CIFAR-10, image-text retrieval and unsupervised object localization.
Our work is related to several strands of research on visual pretraining without full-supervision.
Vision-Language Pretraining: Research on learning visual representations by using textual labels or annotations has a long history. In (Quattoni et al., 2007), the authors learn data-efficient image representations using manifold learning in the weight space of classifiers trained to predict tokens in image captions. Following this work, (Joulin et al., 2016) used convolutional neural networks to predict words in image captions to learn image representations. This approach was later extended in (Lei Ba et al., 2015) where the model learns to predict phrase n-grams, which demonstrated impressive zero-shot performance on downstream classification tasks. Recently, VirTex (Desai and Johnson, 2021) used proxy language modeling tasks, such as image-captioning to train a visual encoder and a transformer based language decoder which generates captions. ICMLM (Sariyildiz et al., 2020) demonstrated a similar masked language modeling approach but relied on pretrained textual encoders for generating textual features. In (Stroud et al., 2020), video representations are learned using paired textual metadata, however the method does not extend to visual pretraining for images. In general, these methods distill the rich semantic information from a caption into the visual representation by learning to predict each token in the caption given the corresponding image. More recent work, such as CLIP (Radford et al., 2021), has shown that a simpler contrastive objective for aligning image and caption pairs is also able to learn a powerful visual representation. Our work extends CLIP using a more information-efficient approach.
Contrastive Representation Learning and Mutual Information Estimation: As demonstrated in (Wu et al., 2020), we observe that contrastive frameworks learn by maximizing the mutual information (MI) between different views of a given data point. For images, this is achieved by maximizing the MI between different augmentations of the data as in SimCLR (Chen et al., 2020a;Bachman et al., 2019). While for sequential data such as conversational text, consecutive utterances can be considered as different views (Stratos, 2018). Similarly, several other contrastive frameworks have been proposed that learn representations in domains such as images (Grill et al., 2020;Caron et al., 2020), text (Mikolov et al., 2013;Stratos, 2018), graphs (Veličković et al., 2018), and videos (Jabri et al., 2020). The value of mutual information is extremely challenging to estimate, especially for the high-dimensional continuous representations used in deep learning. To this end, various tractable lower-bounds on mutual information are used for optimization. Recently, MINE (Belghazi et al., 2018) proposed a general-purpose parameterized neural estimator of mutual information. It uses a Donsker-Varadhan (Donsker and Varadhan, 1983) representation of KL-divergence as the lower-bound on mutual information. MINE (Belghazi et al., 2018) used a neural network critic to distinguish positive and negative pairs of samples. Another popular bound on mutual information that has seen wide adoption due to its low variance is the InfoNCE (Oord et al., 2018) bound. In (Hjelm et al., 2018), the infoNCE bound on the mutual information is used for unsupervised representation learning. While it is used by several other methods for self-supervised (Chen et al., 2020a) representation learning for images. The capacity of the bound is limited by the number of contrastive samples used (McAllester and Stratos, 2020). Additionally, In-foNCE can underestimate large amounts of true MI which is generally the case with high-dimensional representations of natural images. To this end, DeepInfoMax (Hjelm et al., 2018) proposed using a lower-bound on mutual information that is based on the Jensen-Shannon Divergence (JSD) instead of the traditional KL-divergence (KLD). The authors show that the JSD based lower bound is stable, differentiable, and can be optimized with just one negative sample. Inspired by this, we extend the use of this bound for vision-language pretraining and demonstrate its effectiveness through extensive experimental evaluations.
Given a dataset of image-caption pairs, the goal of our pretraining framework is to train an image encoder and a text encoder such that representations learned from the visual and the textual streams share maximum information (Figure 2 shows an overview). Consider an image encoder network, f i with parameters θ i and a textual encoder, f t with parameters θ t . Let (x i , x t ) be a sampled image-caption pair from the dataset and f i (x i ) and f t (x t ) denote the representations extracted from the networks. Based on the information bottleneck principle (Tishby and Zaslavsky, 2015), the maximum mutual information (MI) predictive coding framework (Oord et al., 2018;Hjelm et al., 2018;McAllester and Stratos, 2020) aims to learn representations that maximize the MI between inputs and representations.
In recent years, several methods (Chen et al., 2020a;He et al., 2020;Bachman et al., 2019) have used this principle to maximize MI between representations extracted from multiple views of a shared context. In the case of visual self-supervised learning, this is achieved by creating two independently-augmented copies of the same input and maximizing the MI between the respective features produced by an encoder. This framework can be extended further by considering an image x i and its caption x t as distinct views of the same input. This setup is motivated by the observation that image captions contain rich semantic information about images, for instance, presence of objects, location of objects, their relative spatial configurations, etc. Distilling this information into our visual representation is useful for robust representation learning (Radford et al., 2021). To this end, we formulate our objective as follows:
( θi , θt ) = arg max θi,θt
where I(f i (x i ), f t (x t )) ≤ I(x i ; x t ); due to the data processing inequality between visual and textual streams.
For given random variables y and z, their mutual information is defined as a Kullback-Leibler (KL) divergence between their joint distribution p(y, z) and the product of their marginal distributions, p(y)p(z) as,
However, mutual information is notoriously hard to estimate for high-dimensional continuous variables, especially when the distributions p(y, z), p(x), or p(z) are not explicitly known. As a result, recent approaches use various tractable lower bounds on the mutual information which are differentiable and hence can be maximized with gradient-descent based optimization. For contrastive learning, a commonly used bound is infoNCE (Oord et al., 2018) based on Noise-Contrastive Estimation (Gutmann and Hyvärinen, 2010). This bound is relatively more stable and has been shown to work in a wide variety of tasks (Chen et al., 2020a;Bachman et al., 2019;Chen et al., 2020b) including CLIP (Radford et al., 2021) which, similar to our method, aims to learn visual representations from textual annotations. The infoNCE bound has seen wider adoption as it demonstrates lower variance compared to the Donsker-Varadhan bound (Donsker and Varadhan, 1983). However, both of these bounds require a large number of negative samples and as a result, recent methods either train with extremely large batch-sizes (Radford et al., 2021;Chen et al., 2020a); or an additional memory-bank of negative samples (Chen et al., 2020b;Tian et al., 2020).
Unlike these works, we estimate mutual information using a Jensen-Shannon Divergence (JSD) bound, similar to formulations used for generative modeling (Nowozin et al., 2016); and source separation (Brakel and Bengio, 2017). This bound on mutual information is derived by replacing the KL-divergence in equation 2 with the Jensen-Shannon divergence (ref. appendix for further discussion). Interestingly, the lower bound derived as such is stable, differentiable, monotonically related to the mutual information I(y; z), and most importantly, not dependent on the number of negative samples. Hence we have,
and T ω : Y ×Z → R is a discriminator neural network with trainable parameters ω which are jointly optimized to distinguish between a paired-sample from a joint distribution (positive image-caption pair) and one pair from the product of marginals (negative image-caption pair). Therefore we are able to optimize our overall objective with just one negative sample as follows:
where the visual encoder is a convolution neural network, and features are extracted from the pre-classification layer of the network. The textual encoder is parameterized by a neural network that takes the caption as a string of textualtokens and generates a one-dimensional representation.
In this section, we describe the experiments that demonstrate the value of using textual captions for learning visual representations using CLIP-Lite. In our experiments, the CLIP-Lite architecture consists of a ResNet-50 image encoder and the BERT-base textual encoder and is trained on the COCO Captions (Chen et al., 2015) dataset. We evaluate the robustness of our visual encoder through the following downstream tasks which use the visual encoder (1) as a frozen feature extractor, or (2) as source of weight initialization for finetuning (ref. appendix). In addition, we also demonstrate the data efficiency of our method by evaluating performance on fractional datasets.
In all experiments, we use a standard ResNet-50 (He et al., 2016) that takes in a 224 × 224 image and generates 2048dimensional features at the pre-logit layer. For textual encoding, we use a transformer (Vaswani et al., 2017) model initialized using BERT base (Devlin et al., 2018) and use the output [CLS] token as the text representation. We use the COCO Captions dataset (Chen et al., 2015) which has 118K images with five captions per image. During training time we apply (1) random cropping, (2) color jittering, (3) random horizontal flips while interchanging the words 'left' and 'right' in the caption, and (4) normalization using the ImageNet image mean. We use SGD with momentum 0.9 (Sutskever et al., 2013;Polyak, 1964) and weight decay 10 -4 wrapped in LookAhead (Zhang et al., 2019) with α = 0.5, and 5 steps. We perform distributed training across 8 GPUs with batch normalization (Ioffe and Szegedy, 2015) per GPU with an overall batch size of 1024 images for 250K iterations. We use linear learning rate warmup (Goyal et al., 2019) for the first 10K iterations followed by cosine decay (Loshchilov and Hutter, 2016) to zero. Additionally, we train CLIP (Radford et al., 2021) on the COCO-dataset using an open-source implementationfoot_0 with the originally recommended (Radford et al., 2021) training schedule that suit smaller datasets, reasonable batch-sizes, and compute resources. Specifically, we train using the Adam Optimizer (Kingma and Ba, 2014) with decoupled weight decay regularization (Loshchilov and Hutter, 2016) for all weights except gains or biases.
We train with a batch-size of 1024 and warm-up to an initial learning rate of 10 -4 in 10K steps and decay to zero with the cosine schedule. We found that the performance slightly improves with longer training therefore we train for 250K iterations, similar to ours. All other training details and hyper-parameters were kept the same as the original work (Radford et al., 2021). Please note that our ResNet-50 based CLIP-COCO model outperforms (+1.2% Zeroshot Acc. on CIFAR10) publicly available weightsfoot_1 , refer to appendix for further details on CLIP-COCO training.
As described in main paper, our JSD-based lower-bound on mutual information relies on a discriminator function, T ω : Y × Z → R, which distinguishes between samples extracted from the joint distribution, P (Y, Z) i.e. a positive image-caption pair and the product of marginals, P (Y )P (Z) i.e. a negative image-caption pair. This discriminator function can be modelled as an arbitrary neural network with parameters ω that can be jointly optimized with the encoders during training (Belghazi et al., 2018). In this work, we use a projection and alignment based architecture similar to the one presented in Deep In-foMax (Hjelm et al., 2018).
Given a pair of input one-dimensional representations, both vectors are first projected using a projection module with two linear layers separated by a ReLU and a linear shortcut.
A dot-product of these projections is then computed to get alignment scores. The projection function maps these representations to an aligned cross-modal latent space. Separate projection functions are used for image and text representations. Positive and negative pairs of image-text representations are passed through the discriminator to get respective scores which are then used to estimate and maximize mutual information using our objective. This architecture, in addition to being simple and computationally inexpensive, also offers alignment of the representations into a common cross-modal latent space which uses cosine similarity as the distance metric.
In these experiments, we train linear models on frozen visual backbones pretrained using CLIP-Lite and compare with other pretraining methods on PASCAL VOC (Everingham et al., 2010) and ImageNet-1k (Russakovsky et al., 2015) classification problems.
PASCAL VOC linear classification: For this experiment, our setup is identical to VirTex (Desai and Johnson, 2021). We train on VOC07 trainval split (9K images, 20 classes) and report mAP on the test split. For classification, we train per-class SVMs on 2048-dimensional global average Imagenet-1k linear classification: For this experiment, our setup is identical to VirTex (Desai and Johnson, 2021). We train on the ILSVRC 2012 train split and report top-1 accuracy on val split. We train a linear classifier (fully connected layer + softmax) on 2048-dimensional global average pooled features extracted from the last layer of the visual backbone. For training, we use a batch-size of 256 for 100 epochs. We use SGD with momentum 0.9 and weight decay 0. The learning rate schedule is decayed by 0.1 after 60 & 80 epochs with an initial LR of 30.
Results: We compare CLIP-Lite to supervised, selfsupervised and textually-supervised models in Table 1. CLIP-Lite significantly outperforms baseline CLIP when trained with the same amount of data on both tasks. When compared to other image-caption pretraining methods, CLIP-Lite performs competitively with VirTex (Desai and Johnson, 2021) on VOC2007 and outperforms both VirTex (Desai and Johnson, 2021) and ICMLM (Sariyildiz et al., 2020), which are trained on relatively complex language modeling tasks, on Imagenet classification. In addition, different from them, our method also generates a shared latent space that encodes both image and text modalities and enables cheap computation of cross-modal alignment, which enables additional downstream tasks such as zero-shot retrieval, and zero-shot transfer. It also allows us to find subspaces associated with abstract concepts that are better expressed with language than with visual examples, which allows for applications in bias mitigation through the synthesis of gender-neutral image representa-
Next, we evaluate the performance of of our visual backbone when the entire network is finetuned for the downstream task. For this purpose, we perform fine-grained classification on the iNaturalist 2018 (Van Horn et al., 2018) dataset, which contains images from 8, 142 finegrained categories, with a long-tailed distribution. We train with the 'train2018' split and evaluate in the 'val2018' split. We finetune pretrained ResNet-50 models with a linear layer, using SGD with momentum 0.9 and weight decay 10 -4 for 100 epochs. Initial learning rate is set to 0.025, which is reduced by 10× at epochs 70 and 90. We use a batch size of 256 distributed across 8 GPUs.
Results: We summarize our results in Table 3. CLIP-Lite is competitive with supervised and self-supervised learning models trained with images alone even those trained with 5-10x more images. Its performance matches closely a model trained with full-supervision on 50% of the Ima-geNet (Krizhevsky et al., 2012) dataset, equal to 5.4× the number of images as our pretraining dataset. Finally, CLIP-Lite obtains a 1.3% improvement over CLIP-COCO, while being competitive with VirTex.
Our method is expected to produce effective representations for the task of image-text retrieval as it is trained by aligning text and image representations. We evaluate the image-text retrieval capabilities of CLIP-Lite on the validation set of COCO and the test split of Flickr30k (Young et al., 2014) datasets, following CLIP. We perform zeroshot image-text and text-image retrieval by ranking imagetext pairs by their alignment score, which is the dot product of the normalized representations in the shared latent space. This ability to perform zero-shot retrieval is a salient feature of our and CLIP-like methods over previously proposed works that rely on language modeling tasks.
Results: Table 4 shows that CLIP-Lite substantially outperforms CLIP-COCO on all metrics for both text and image retrieval. The performance improvement is large both when evaluated on the COCO validation set, which is similar to the the COCO-Captions training split used for CLIP-Lite training; and when testing zero-shot on unseen text vocabulary and object categories of Flickr30K. Taken together, these results show that CLIP-Lite learns a superior representation for retrieval tasks as compared to CLIP, when trained on same amounts of data.
We use the cross-modal alignment capability of CLIP-Lite to perform zero-shot classification on unseen datasets Table 4: Retrieval Results: CLIP-Lite substantially outperforms CLIP-COCO and the baseline Visual N-grams (Li et al., 2017) approach. CLIP-Lite is superior when evaluated on the COCO test split, which is similar to the CLIP-Lite training set and on Flickr30K, generalizing to unseen images and text in a zero-shot manner.
Flickr30k MSCOCO Flickr30k MSCOCO Method R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10
Visual CIFAR-10, CIFAR100 (Krizhevsky et al., 2009), Ima-geNetV2 (Recht et al., 2019), and ImageNet-A (Hendrycks et al., 2021). Our model generates a shared latent space where we can readily compute the alignment between given (image, text) pairs as the cosine similarity of their representations. Therefore, we use the names of the classes to generate a textual description of each class label (class prompt). In this experiment, we use templates such as, "a photo of a {class name}" to generate such class prompts, following CLIP (Radford et al., 2021). Please refer to the appendix for comparison between different templates for generating the prompts. For a given image, we compute its alignment with each of the class prompts which are then normalized into a probability distribution via a softmax.
Results: Our results for the zero-shot transfer task on unseen datasets are compiled in table 5. Given the zero-shot nature of the task, CLIP-Lite obtains satisfactory performance on the complex ImageNet evaluations while clearly outperforming CLIP trained with the same amount of data in all settings.
Next, we evaluate the capability of CLIP-Lite to localize a region in the image that corresponds to a given textual description. We compute the dot-product of the visual and textual embedding and compute its gradients with respect to the last convolutional layer of ResNet. We global average pool these gradients and perform a weighted sum with the last convolutional activations and clip the negative values to obtain Grad-CAM (Selvaraju et al., 2017). We then use the areas highlighted by Grad-CAM to approximate a predicted bounding box. We evaluate this experiment on the RefCOCO+ (Yu et al., 2016) dataset. We note that the images in the RefCOCO+ dataset are extracted from the training set of the COCO (Chen et al., 2015) dataset which our model uses for pretraining. Therefore, we view this evaluation as an explorative study to establish that our model is focusing on the relevant areas of the image while computing the alignment score with the caption.
RefCOCO+ results can be seen in the table to the right. CLIP-Lite significantly outperforms CLIP on all settings. Qualitative results in Figure 4 demonstrate that even though the network has not been trained with any localization supervision, it is surprisingly good at localizing phrases in the image. For instance, in Figure 4 bottom left, for the phrase "blue", the network attends to all blue regions in the player's outfit. Interestingly, it is also able to localize abstract concepts as "blurry player".
One salient feature of CLIP-like methods, which other methods such as VirTex (Desai and Johnson, 2021) and ICMLM (Sariyildiz et al., 2020) lack, is that they are able to generate a shared latent space that encodes both image and text modalities. This enables us to find representations and subspaces associated with abstract concepts that are better expressed with language than with visual examples. Using this property, we demonstrate a methodology to remove concepts from visual representations. For instance, it is non trivial and even problematic to collect visual examples that capture the concept of gender, while it is relatively straightforward to express this concept in a sentence using language. Therefore, we can identify the gender subspace in our shared embedding space using text and use it to remove variance along this direction to smooth out the concept of gender from image representations. We "bending over" "child" "man" "girl"
"blurry player" "blue" "red bus" "grand bazar blue"
Figure 4: Visual Grounding on RefCOCO+: CLIP-Lite is able to localize textual descriptions to relevant areas in the image, shown here through Grad-CAM visualization using the alignment score with the mentioned textual description. Top left: CLIP-Lite is able to localize the action phrases such as "bending over". This demonstrates the value of learning from semantically rich textual captions.
motivate this experiment in the growing body of literature regarding bias mitigation, where the objective is to build invariant representations with respect to sensitive or protected attributes (Wang et al., 2019(Wang et al., , 2020)). In comparison to our work other methods require retraining the models to obtain invariant bias representations through adversarial learning (Wang et al., 2019) or effectively combining domain independent classifiers (Wang et al., 2020).
Identifying the Concept Subspace: The first step of our approach is to isolate the direction in the embedding space that captures maximum gender variance. For this purpose, we follow a strategy similar to Bolukbasi et al. (Bolukbasi et al., 2016) that deals with debiasing word representations. For characterizing for male and female genders, we use word pairs (man, woman), (son, daughter) that indicate opposite genders. Now, consider a dataset D = {(w m , w f )} m i=1 where each entry (w m , w f ) is a tuple of opposite gendered words. Intuitively, each tuple should contain words that have the same meaning if not for the target attribute. To make the set D more robust, we used the sentence contextualization strategy presented in Liang et al. (Liang et al., 2020). In this step, the predefined sets of gendered tokens in the set, D, are used to generate paired sentences which have the same meaning except for the gender attribute. We perform this contextualization by using simple sentence templates such as "I am a [word]" where [word] can be replaced with the word pairs in our dataset D to give, for instance, ("I am a boy.", "I am a girl."). Hence, we obtain a contextualized bias attribute dataset S = {(s m , s f )} n i=1 where each entry is a tuple of semantically similar sentences with opposite genders. We extract the sentence representations for all entries in the set S by passing them through our pretrained text Table 6: Concept Editing Results: We compute the mean alignment scores for the top 10 images queried using prompts that either contain male or female gendered tokens. The images are queried using gendered and neutralized representations. We observe that after gender-deletion the alignment score for images with men and women converge to similar values.
Images with Women gendered neutral delta gendered neutral delta Male queries 0.085 0.069 +0.016 0.057 0.067 -0.010 Female queries 0.042 0.068 -0.026 0.089 0.062 +0.027 encoder and then projecting them to the shared latent space using the projector trained with our mutual information discriminator T ω . We define sets R m and R f that contain sentence representations of the male and the female category, for example, R m = {F t (s m )} n i=1 where F t (.) is the sequential combination of our pretrained text-encoder and text-projection functions. Now we estimate the gender subspace V = {v 1 , ..., v k } using the Principal Component Analysis corresponding mean shifted representation from both sets as described in (Liang et al., 2020).
Removing Concept from Image Representations: After estimating the gender subspace in our shared cross-modal latent space, we extend the hard debias algorithm (Bolukbasi et al., 2016) to edit visual representations. This is achieved by first projecting the representation onto the bias subspace, this projection is then subtracted from the original representation to give the de-gendered representation. Given an image, we first encode the image onto our multi-modal shared latent space to get, say, h. Now, consider the identified gender subspace V , we first com-Gendered Neutralized "A woman with a cellphone" "A man in a store" pute the projection of h onto this gender subspace V to get h V = k j=1 ⟨h, v j ⟩ v j . We subtract this projection from the original representation to get a vector, ĥ = h -h V that is orthogonal to the bias subspace and therefore does not encode the target bias.
Analysis: To evaluate concept editing, we use the gendered subset of COCO-Captions (Wang et al., 2019;Zhao et al., 2017) for studying bias. The gender labels for images in the COCO dataset are derived from the captions. We obtain a subset from the COCO dataset with 16, 225 images with men and 6, 601 images with women. We use 10 sentences with male references and 10 sentences with female references from the set S and use them as prompts for this study. For each gendered prompt, we query the top 10 images independently from the male and the female image sets using both biased and debiased representations to compute alignment with the prompt. The mean alignment scores are then computed for each set given the prompt. Table 6 shows that the alignment scores roughly equalize for members of the two groups after removing the variance along the gender direction from the visual representations which indicates the invariance of the visual representations to gendered language tokens.
CLIP-Lite trains the visual encoder by maximizing the mutual information between images and their captions. We observe that language supervision provides rich semantic density which can be distilled into visual representations. The visual encoder is encouraged to learn visual representations that encode maximum information from captions. As such, the visual encoder is only aware of concepts and objects that human-annotators have mentioned in the captions. Therefore the visual encoder lags behind task-specific models that are trained specifically for a given fine-grained task. For instance, visual encoders trained with CLIP-Lite struggle with relatively contextual downstream tasks that involve reading text or counting number of objects in an image. In this work, we train CLIP-Lite on the COCO-Captions (Chen et al., 2015) dataset which has high-quality curated captions for images. However, when trained on datasets with text paired with images from the internet, the textual captions can be significantly unfiltered and noisy. Our method essentially learns by aligning the caption and text representations. Therefore, the model is susceptible to learning harmful biases that are represented in the captions. Hence, deployment of visual backbones trained with CLIP-Lite and other pretraining methods which use natural language supervision need to be analyzed specifically for such biases. In this work, we present an approach to edit concepts from visual representations using the shared vision-language latent space learnt by our method. For instance, we demonstrate this capability by editing visual representations such that they are invariant to gendered tokens in language. However, further explorations are required to develop this concept editing mechanism further.
We introduced CLIP-Lite an image-text pretrained model using contrastive learning that leverages a different objective than the CLIP model that allows for it to be more data efficient. CLIP-Lite's objective is insensitive to the number of negative samples and hence can be trained with just one negative image-caption pair and shows superior results on lower data regimes while still demonstrating some of the most remarkable capabilities of the original CLIP model such as transferable features, zero-shot capabilities, and a shared latent space. Additionally, we present a concept editing methodology for neutralizing visual representations with respect to a chosen abstract concept. Please refer to the supplement for a detailed discussion on limitations and potential impact of our approach.
distributed training across 8 GPUs with batch normalization (Ioffe and Szegedy, 2015) per GPU with an overall batch-size of 1024. We warm-up to the initial learning rate in 10K steps and decay to zero with the cosine schedule. We found that using the learning rate of 10foot_3 works slightly better (+1.4% on VOC07) than the originally recommended 5 × 10 5 . We also found that the performance incrementally improves (+1.9% on VOC07) with longer training therefore we train for 250K iterations, similar to ours. All other training details and hyper-parameters were kept the same as the original work (Radford et al., 2021). Please note that the ResNet-50 backed CLIP model trained by us on the COCO dataset outperforms (+1.2% Zero-shot Acc. on CIFAR10) publicly available weights 4 .
https://github.com/mlfoundations/open_clip
https://github.com/revantteotia/clip-training/blob/main/ zero_shot_eval_output/coco_trained_clip_observations.md
https://github.com/mlfoundations/open_clip
https://github.com/revantteotia/clip-training/blob/main/ zero_shot_eval_output/coco_trained_clip_observations.md