Humans take much longer time to name a colored word when the color and the word mismatch (e.g., "red" shown in green) than when they match (e.g., "red" shown in red) [1]. This effect is an example of rich psychological evidence suggesting that humans learn language by grounding meanings to knowledge about the world [2,3]. In contrast, most models in natural language processing (NLP) [4][5][6][7] encode "distributional semantics" [8] learned from texts only. Put yourself as machines in a thought experiment for the "Chinese Room Argument" [9]. Imagine that you have to learn Chinese from scratch as your first language. All that you have is a Chinese-to-Chinese dictionary. You might be able to relate a word to other words based on textual distributions. It is, however, impossible to learn word meanings without any additional explanation in reference to the physical world [10].
A language model may learn concepts from texts paired with sensory data, such as images. Joint vision-language learning has been explored for image captioning [11], visual question answering [12], and pre-training vision models with weak supervision [13,14]. In line with these studies, we train a language model and a vision model jointly to match images and texts. We further analyze the semantic space obtained with the visually grounded language model. In this space, semantic embeddings are found to be organized and clustered by visual attributes, predictive of human-defined norms of semantic features, useful for compositional language understanding and cross-modal image search. We expect this visually grounded language model to also be useful for understanding the computational basis of grounded cognition [15,16].
Figure 1: Visual grounding of natural language (see Section 3.1). The visual and language streams take an image and its caption as input, respectively. The inner-product between the visual feature maps and the contextual word embeddings forms the 3D match-map that highlights the matching between visual and language content. The similarity score calculated from the match-map (see Eq. 1) is used to evaluate the cross-modal contrastive loss.
2 Background and Related Work
In the distributional hypothesis [17], words that occur in similar contexts carry similar meanings. This hypothesis has motivated influential machine learning models to learn word embeddings from large text corpora [4,18]. However, the learned word embeddings are not straightforward to interpret [8,19]. Alternatively, the symbol grounding hypothesis suggests that a word is connected to its meaning by relating to its referent in the physical world [10,20]. In line with this hypothesis, earlier studies demonstrate that visual features or contexts can enhance language learning [21][22][23][24][25].
Grounding language in vision has been of increasing interest in computational linguistics and machine learning. A common strategy is to fuse words with related visual information in terms of perceptual norms [21], bag-of-visual-word [23,26], or learnable visual features [27][28][29][30]. The models used for vision-language fusion evolve alongside those for NLP, such as Latent Dirichlet Allocation (LDA) [23], log-bilinear model [31], Skip-gram model [25,27], and recurrent neural network [28]. Generally, visual grounding may refine the distribution and interpretability of language representations [23,25,26,[28][29][30][31][32] and facilitate cross-modal tasks [28][29][30][31][32][33]. More recent work has begun to use transformer [6,34] for vision-language learning, showing strong performance in cross-modal tasks [35][36][37][38] .
Contrastive learning [39,40] is increasingly applied to not only unimodal data [41] but also multimodal data [13,42]. It is able to learn better representations than alternative prediction or classification objectives [43]. However, cross-modal contrastive learning is still under-explored for higher-level tasks, e.g., visual question answering [12], visual reasoning [44], scene graph generation [45]. Such tasks involve abstract reasoning about the relations between entities (e.g., visual objects). Prior work approaches relational inference with multi-layer perceptron [46,47] or graph neural networks [48][49][50]. Arguably, a more compelling idea [51] is to model entities as vectors in a continuous space and to model their relations as arithmetic operators (linear [52][53][54] or bilinear [55,56]) applied to the vector representations of those entities.
In this work, we first build a two-stream model to jointly learn visual and language representation from image-caption pairs, similar to recent work [13,14]. We then finetune the learned model by adding a cross-modal attention layer [35,36] and bilinear operators [55] to represent the relations between visual objects. Both stages utilize cross-modal contrastive loss. Related to our work, Harwath et al. match visual objects to spoken words using triplet loss [42]. Early this year, Jia et al. [13] and Radford and Kim et al. [14] use contrastive learning to pretrain a vision model using a massive image-text dataset and demonstrate largely improved zero-shot transfer learning performance on visual and cross-modal tasks. Different from their perspectives, we focus on assessing the language encoders and word representations. Specifically, we perform a systematic evaluation of the semantic space grounded in vision vs. the ungrounded semantic space learned from texts only. This evaluation is possible since after training, the language and visual streams in our model are fully separable as stand-alone systems, unlike some vision-language models that require both visual and textual input to be usable [35,36]. Our goal is to assess how visual grounding affects the distribution of textual representations by analyzing the distribution of word embeddings in the grounded semantic space, in line with related works [23,25,26,[28][29][30][31][32]57] .
To build a computational model for learning visually grounded language representations, we develop a model (Fig. 1) that combines a stand-alone visual stream and a stand-alone language stream. The visual stream is based on VGG16 [58] with an additional linear transformation as an embedder to match the feature dimension of the language stream and an additional multi-head self-attention layer [59] to enforce global information aggregation and learn long-range dependency. The language stream is based on Bert [6]. Using separate linear transformation heads [41], the output from both the visual stream and the language stream are projected to a common representational space. In this common space, the inner-product between the visual representation V at every location and the language representation L of every word gives rise to a 3D match-map, where each element indicates how a word in the text matches each location in the image (See illustration in Fig. 1). The sum of the maximal match is the similarity score S(V , L) between a pair of image and text. See Eq. 1, where i, j indicate the location in the 2D image feature map V and k indicates the k-th word in L.
Extending the unimodal normalized temperature-scaled cross-entropy (NT-Xent) loss [13,41,60], we define the cross-modal contrastive loss using the anchor sample from one modality and the positive sample and negative samples from the other modality [13,14]. As such, we define and sum two loss functions with the anchor sample from either images or texts and positive/negative samples from either texts or images, respectively.
For Loss l in Eq. 2, the anchor sample V i is an input image and the positive sample L i is the corresponding image caption, whereas the negative samples L j are unmatched textual descriptions included in the same batch (B is the batch size). Similarly, Loss v in Eq. 2 is defined to contrast the positive and negative image samples against an anchor textual sample.
We further finetune the model for visual relation prediction, as illustrated in Fig. 2. In this stage, we remove the linear transformation heads in Fig. 1 and add a multi-head cross-modal attention module [35,36]. The attention module uses a query based on the embedding of an object word from the language stream (Query L ) and uses keys (Key V ) and values (Value V ) from every location in the feature map output from the visual stream. The attention score is calculated as the inner-product of Query L and Key V followed by softmax. The attention-weighted sum of Value V is concatenated across 8 attention heads to generate a visually grounded object representation.
For visual relation prediction, we also use contrastive learning with two loss functions by taking either relation embedding or subject/object representations as positive/negative samples.
In Loss rel , K rel is the set that contains all relations available. The anchor sample is a pair of subject and object in an image. The positive sample is the embedding of the ground truth relation. The negative samples are the embeddings of all other relations. In Loss obj , the anchor sample is a given relation. The positive sample is a subject-object pair that holds this relation. The negative samples are other subject-object pairs in a different relation. For both loss functions, the positive and negative samples are drawn from the same batch B.
In addition, we also add a classification head (two fully connected layers with ReLU in between) and apply it to the grounded object representation. We use object classification as an auxiliary objective (with a cross-entropy loss) to constrain the grounded object representation to be separable across objects for classification.
We train the model in three stages to progressively refine the model with increasingly demanding tasks.
In the first stage, we pretrain the visual and language streams separately as image and text encoders. The language stream is the pretrained Bertfoot_1 used as the baseline model for subsequent experiments. The visual stream is pretrained for object classification with ImageNet [61]. Relative to the baseline CNN, the inclusion of self-attention improves the top-1 classification accuracy from 71.6% to 74.3% on the ImageNet validation dataset. The attention module also renders the classification more robust when the input image is partially occluded (See details in Appendix A.1).
In the second stage, we refine the pretrained language and visual streams by matching texts to images, as illustrated in Fig. 1 on the MS COCO dataset [11]. While freezing other layers, we refine the self-attention layer in the visual stream and the top k layers in Bert (by default k = 8). Training with contrastive learning is based on the MS COCO dataset. As five captions are available for each image, we randomly sample one caption per image in each iteration. Earlier grounding (larger k) tends to support better image-text retrieval performance (see details in Appendix A.2).
In the third stage, we further finetune the model for visual relation prediction as illustrated in Fig. 2.
We refine the visual self-attention layer and the higher l layers in Bert (by default l = 2) based on the Visual Genome dataset [45]
To evaluate the visually grounded semantic space, we use the language stream as a stand-alone model to extract the output representations of commonly used English words in the SemCat dataset (9,197 words; 100 word categories) [62]. Details about how word representations are extracted from the language stream are explained in Appendix B. We apply the principal component analysis to the representations of all the words studied here and examine the top components as the principal dimensions of the grounded semantic space. Interestingly, the first principal dimension is readily interpretable as an abstract-to-concrete axis (Fig. 3). For example, words with the highest values in this axis are ostrich, seagull, albatross, blender, pelican, broccoli, parakeet, lettuce, sailboat, vegetables, whereas words with the lowest values are displeasure, liking, to, outgoing, present, experienced, profitable, faithful, meaningful, multitude.
The representations of words along this axis is significantly correlated with human rating of their concreteness (ranging from 1 to 5) from prior study [63] (Fig. 3). The Pearson correlation coefficient reaches 0.8749 or 0.6615 across word categories or individual words, respectively; after grounding with object relations: r = 0.8001 for categories, r = 0.6948 for words. In contrast, the principal axis of the ungrounded semantic space learned from the baseline Bert model is not straightforward to interpret and shows a weak correlation with human ratings of concreteness (Table . 1). Other principal components are also intuitively interpretable. For example, PC 2 captures the human vs. non-human axis, PC 3 captures the scene vs. object axis, PC 4 captures the natural vs. artificial axis, PC 5 captures the indoor vs. outdoor axis, PC 6 highlights words related to food. See results about other principal components in Appendix B.1.
We further ask whether the visually grounded word embeddings are amenable to binary semantic features defined by humans [25,64]. We use the concept property norm dataset from the Centre for Speech, Language and the Brain (CSLB) [65]. The dataset includes binary semantic features (e.g., has_wheels) labeled for 638 concepts collected from 123 human participants. We keep 390 features that each contains at least 5 samples. We hypothesize that the grounded word embeddings can be readout with a linear and sparse projection to readily support binary classification attainable by humans. To test this hypothesis, we train a logistic regression model with L1 regularization to predict each binary semantic feature from the grounded word embeddings and also repeat this for ungrounded semantics for comparison. See Appendix B.2 for details about this dataset and our evaluation method.
Results suggest that the grounded word embeddings are significantly more predictive of visually relevant binary features than ungrounded counterparts obtained by Bert (Wilcoxon Signed Rank Test; p < 0.0001) (Fig. 4). This difference is less pronounced but still significant for other features related to other perceptual (e.g., has_flavors), functional (e.g., does_cut), encyclopaedic (e.g., is_dangerous), and taxonomic features (e.g., is_clothing), especially after visual grounding of object relations.
After visual grounding, the semantic representations tend to group themselves based on perceptual similarity. We use the SemCat dataset (9, 197 English words from N = 100 categories) [62] and calculate the Silhouette coefficient (between -1 and 1) to measure the degree to which these words are clustered by categories. The distance between word embeddings is measured as the cosine distance (See details in Appendix B.3). The Silhouette coefficients across 100 categories are significantly higher for the visually grounded semantics than ungrounded ones (Wilcoxon Signed Rank Test; p < 0.0001) (Fig. 5 left). The greatest gain in clustering are noticeable for categories that include concrete concepts (e.g. car, housing, mammal) with defining visual attributes (Fig. 5 right). For some abstract categories related to human emotion (e.g., happy), the grounded representations are also better clustered than the ungrounded ones.
A drawback of distributional semantics is the inability to make visually informed compositional reasoning. We know that "zebra is a horse with black and white stripes", because we have seen how zebra looks like, whereas an ungrounded language model is never or rarely exposed to such The right part shows the corresponding results after visual grounding of natural language. Orange lines indicate words with increased ranking after visual grounding and blue lines for the decreased cases. We highlight in red the target word "zebra" for this specific example, which shows a significant increase in cosine similarity (from 0.12 to 0.60) and ranking (from 2914 to 12 out of 6238 unique words). Besides, the top words similar to "striped horse" are all horse-like animals after visual grounding, but this is not the case for the ungrounded Bert model.
information [10]. We test whether the visually grounded semantics can perform compositional reasoning based on visual knowledge, without being explicit trained to do so. We choose some words (Table 2), for which the meaning can be intuitively inferred from the combination of other words.
Table 2: Examples of visually informed conceptual composition. Each row shows the cosine similarity and its ranking in the vocabulary (unique words in the Semcat dataset) between the query phrase and the target word. Except (hot weather, summer), all others are concepts supported by composition of visual knowledge in the query phrase. For each case, the highest similarity are rank are in bold. For example, we use a phrase "striped horse" as a compositional query to search for the matched words ranked in terms of cosine similarity. In Fig. 6, the left part shows the cosine similarity and ranking between each of the listed words and the query phrase "striped horse" before visual grounding. The right part shows the corresponding results after visual grounding of natural language. With the grounded semantic representation, the phrase striped horse is highly similar to the word zebra (cosine similarity: 0.60), which is ranked as the 12-th in the vocabulary. After further grounding the language model with visual object relations, the target word zebra has an even higher cosine similarity of 0.63 ranked the 8-th in the vocabulary (Table 2). Other top-ranked words all refer to horse-like animals (i.e., horse, mule, mare, stallion, donkey, camel, antelope). This is in sharp contrast to the ungrounded semantic space, in which it is impossible to relate striped horse to zebra based on the similarity of their representations (cosine similarity: 0.12; rank: 2,914). The ungrounded Bert model highlights the top-3 similar words as tomcat, seahorse, squirrel, which are animals sharing fewer visual features with horse-like animals. See other examples in (Table 2 and Appendix B.4). In our model, the cross-attention module forms a joint representational space to combine both visual and textual input. We explore whether this joint space can be used to support cross-modal tasks, e.g., image search based on image, text, or their combinations [13]. For this task, we add two additional heads (F V and F L ). Each head includes two linear layers with ReLU in between followed by average pooling (See details in Appendix B.5). It is applied to either visual or textual representations in the joint space and results in a single vector representation for an image or a text (Eq. 6, d = 768). While freezing our model described in Section 3.2, we train the two additional heads with contrastive loss to match the average-pooled representations of paired images and texts in terms of their cosine similarity using the MS COCO dataset. To use the model for image search, we apply weighted sum to the normalized representations of a query image and a query text (with weights: 1 -α and α, where 0 ≤ α ≤ 1). We use this multimodal query (Eq. 7) to search a held-out databasefoot_2 for the matched images ranked in terms of cosine similarity.
As α controls the weighting between the textual and visual queries, we test how the image search returns different results as α increases from 0 (image only) to 1 (text only). For example, when we combine a word (horse) and an image (a stripped pattern) into a query, the search finds images similar to the zebra's skin pattern when α is close to 0, or finds images of typical horses when α is close to 1, but not necessarily a zebra for either case until when α is somewhere close to 0.5 (Fig. 7, left). This observation is generalizable to other examples. See similarly graded changes in (Fig. 7, right).
In summary, we apply visual grounding to not only words but also relations between words through cross-modal contrastive learning. The results suggest that grounding language learning in vision renders semantic representations more interpretable by human intuition. The grounded semantic space has its principal dimension encode the concrete-to-abstract variation consistent with human ratings and neurobiological knowledge. The grounded semantic representations are better clustered by finer categories and capable of compositional reasoning (e.g., zebra = striped horse). In addition, our work also shows compelling evidence that both text and image-informed semantics are represented in a common, continuous, and grounded semantic space. Although this notion has been hypothesized in neuroscience and linguistics, it has been rarely implemented and demonstrated with computational models. Uniquely, we demonstrate that a continuously varying combination of a text and an image into a multimodal query can be used to search images, showing results that make intuitive sense.
Several limitations of our work are noteworthy. The datasets used to train our model are orders of magnitude smaller than those used in recent studies [13,14]. Scaling up the model training with increasingly larger datasets is expected to greatly improve the model's performance for cross-modal tasks, while generally preserving the interpretability of the grounded language representations as described herein. Some of our experiments and results are preliminary and primarily for illustrative purposes and await more comprehensive and quantitative evaluation in future studies, especially with more downstream vision-language tasks. Whereas our evaluation focuses on the language model, grounding language to vision may also have refined the visual stream, awaiting further evaluation against visual tasks, as demonstrated in [13,14].
The visually grounded language model may be usable as a computational model for studying the grounded cognition -a theory in cognitive science [15,16]. Ungrounded linguistic models are explanatory about semantic processing in the brain's language network [66][67][68][69][70]. Combining the grounded language model with human behavioral and neural data may elucidate how the language network interacts with distributed sensory and motor areas for semantic processing [71].
It is natural to extend this study by incorporating other sensory input [72,73] and further ground language learning in action [74][75][76][77] and emotion [78]. This study also leaves an open question as to whether grounding should occur at an early or late stage of natural language processing, which awaits further exploration and evaluation. For comprehensive modeling of language grounding, it is desirable to expose an agent to a naturalistic and multi-sensory environment and to engage interactive actions to allow the agent to learn knowledge in the physical world, like how humans learn language.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
bert-base-uncased: https://huggingface.co/transformers/pretrained_models.html
41, 600 images from the validation dataset of Open Images Dataset V 6.