Multi-modal Large Language Models (MLLMs) trained on both text and images are rapidly moving from research into deployment and are being used by millions of people. Yet, while there have been some advances in understanding how Large Language Models (LLMs) work, much less has been done to understand MLLMs. This paper begins to close this gap by studying how information is stored and transferred in MLLMs. To do this through the lens of a factual Visual Question Answering (VQA) task -a very common use case of MLLMs today [26,35,27].
MLLMs process factual information in two steps: information storage and information transfer. Information storage refers to how facts from a pre-training dataset are stored in a model's parameters -its so-called 'parametric' memory. Information transfer describes how information from input Figure 1: MLLMs retrieve information from earlier internal layers compared to their LLM counterparts.
We find that very early MLP layers [1][2][3][4] have high indirect estimation effects to outputs (i.e., they are causal) in LLaVa-7B, whereas the middle MLP layers [4][5][6][7] are causal in LLaMA (Vicuna)-7B. For LLaMA, a larger window size (e.g., 5) is also required to find causal sites, compared to a window size of 1 for LLaVA-7B.
prompts are propagated through these storage locations to the model's final output. Understanding these mechanisms can have many benefits, including ensuring models are factually grounded and informing better evaluation protocols.
Extensive works have explored how LLMs store and transfer factual information [12,29,24], however, this has not been studied for multi-modal inputs. For example, it is suggested that auto-regressive Transformer-based LLMs store factual information in their mid-layer MLP parameters [12,24]. However, MLLMs and LLMs are different: an MLLM involves an additional (continuous) image input, alongside a (discrete) text prompt, and requires additional modules to process it [26,18,19,16,7]. Typically, a vision encoder (e.g. CLIP) is used to convert this image into either visual tokens via a projection layer [18,19,11] or cross-attention layers [3,7,16,36] which are then integrated into the language encoder. These differences suggest that our existing understanding of LLM information storage and transfer may not map directly to MLLMs.
In this work, we use a factual VQA task to study the mechanisms of multi-modal information storage and transfer. We use a constraint-based formulation which views a visual question as having a set of either visual or textual constraints (e.g. What movie directed by this director has won a Golden Globe?). The information retrieved by the model should satisfy these constraints (e.g. this director, Golden Globe) for the answer to be factually correct. This formulation, therefore, offers a systematic way of understanding a model's behavior. Under this framework, we propose MULTI-MODALCAUSALTRACE, which extends LLM causal tracing [12,28,33] to the multi-modal setting, to understand information storage, as well as leverage attention contribution methodologies [37,8] to study information transfer in MLLMs. We also introduce VQA-Constraints, a new dataset of 9.7k factual questions annotated with constraints, spanning natural images (from OK-VQA [22], WikiMovies [37], and Known [12]). With these tools, we study how a widely-used MLLM family processes multi-modal information, specifically LLaVa [18,19] and multi-modal Phi-2 [11]foot_0 .
Our key findings are that MLLMs, in contrast to LLMs: 1) retrieve information from earlier MLP layers (i.e. layers 1-4 vs layers 4-7 in a LLM) (see Fig. 1); and 2) use less parametric memory (require a smaller window size to retrieve this information), when answering a multi-modal question. We also find that information is transferred from a given image to these early MLP blocks through a consistent subset of visual tokens (e.g. last ∼36 tokens from LLaVa's CLIP encoder), and that the self-attention blocks in the middle layers are primarily responsible for shuttling this information to the last token.
Finally, we demonstrate that by editing these early causal MLPs, we can correct errors and insert new factual information into an MLLM. Specifically, we propose a new model-editing algorithm, MULTEDIT, which modifies the projection matrix of the early causal MLPs with a closed-form update. We empirically show MULTEDIT's effectiveness on questions from VQA-Constraints and Encyclopedic VQA [25]. In summary, our contributions are:
1. A novel multi-modal causal tracing methodology that can be used to study information storage from image and text inputs in MLLMs. 2. A new dataset, VQA-Constraints, of 9.7K factual visual questions about natural images annotated with constraints to support future research in these directions. 3. A suite of novel insights on the mechanisms underlying multi-modal information storage and retrieval in MLLMs. 4. A model-editing method, MULTEDIT, which we demonstrate can precisely correct erroneous information and insert new long-tailed information in an MLLM.
Multimodal Large Language Models.. We consider a MLLM to be a model that takes an image and text as input, and generates a text output [2]. Over the last year, such models have made tremendous advances in tasks like VQA and image captioning, including BLIP [15], BLIP-2 [16], Instruct-BLIP [7], LLaVA [18,19], Flamingo [3] and multi-modal Phi-2 (from the Bunny repo) [11]. These MLLMs can broadly be categorized into two families based on how their visual information is integrated into the language model: (i) by embedding the vision encoder's output into each layer of the language model with a cross-attention layer (e.g., Flamingo, BLIP) or, (ii) by mapping the vision encoder's output into "visual tokens" in the language model's input space (i.e. alongside the text tokens) via a projection layer (e.g., LLaVA, Bunny). Both families are widely used, however, the projection layer family has recently shown stronger performance on popular benchmark [18,19,11]. We, therefore, focus our study of information storage and transfer on this model family.
A well-established arm of model interpretability examines the relationship between a model's performance and its internals. A range of recent works have studied the internal mechanisms of information storage [23,34,24] and transfer [9,37] in LLMs. However, to the best of our knowledge, only a few works [30,31] have studied the interpretability of MLLMs, with none specifically investigating the relationship between a model's outputs and its internal states. [30], for example, designs an interactive interface to visualize the attention maps in an MLLM, while [31] explores the shortcomings of the CLIP vision encoder in MLLMs. Neither consider the influence of both vision and text inputs on model internals or offer causal insights, as our work does. Our model editing approach which targets the projection layer MLLM family, is complemented by [6], who propose baselines for inserting information into the cross-attention layer MLLM family.
In this section, we describe the constraint-based formulation we use to study information storage and transfer in MLLMs. Under this framing, we introduce MULTIMODALCAUSALTRACE, a novel causal information tracing technique which we use to study multi-modal information storage. We also
Restoring MLPs Restoring Self-Attention Layers Dataset: Movies Dataset: Multimodal Known Layers Restoring MLPs Restoring Self-Attention Constraint Tokens Constraint Tokens Last Token Last Token Other Tokens Other Tokens Constraint Tokens Constraint Tokens Very early Causal layers Very early Causal layers Very early Causal layers Very early Causal layers Layers Layers Layers Layers Very early Causal layers Very early Causal layers Constraint Tokens Constraint Tokens Other Tokens Other Tokens Dataset: OKVQA Restoring Self-Attention Restoring MLPs describe how we use attention contributions [8] to study multi-modal information transfer. Finally, we describe VQA-Constraints, a new test-bed of visual questions annotated with constraints.
Prior works have used a constraint satisfaction framework to study LLMs [37,20,14] as they offer a systematic way of studying model behavior. This framing defines a constraint as a set of words in the question. The model must retrieve information that satisfies this constraint from its parametric memory in order to generate the correct answer. For example, for the question "What city is the Space Needle in?", the model must retrieve information relevant to the constraint "Space Needle". In the multi-modal setting, we consider these textual constraints as well as introduce visual constraints. We define a visual constraint to be a set of words in the question which refers to an entity in the image. The model must similarly retrieve information about this entity to generate the correct answer. For example, given an image of Christopher Nolan and the question "Name a movie directed by this director in 2006?", the visual constraint is "this director" (and the text constraint is "2006"). We refer to a question involving both a visual and text constraint as a multi-constraint question, and ones with only a visual constraint as a single-constraint question.
We use this framework to study the widely-used "projection layer" MLLMs family. These models are composed of a visual encoder f θ , a large language model g ϕ and a projection head p γ . The projection head p γ is responsible for mapping the output of the visual encoder into the input space of the language model, as so-called "visual tokens". Given an image-text pair denoted as (x, y), the language model processes them as
i=1 is the set of visual token embeddings and t(y) = {t i } M i=1 is the tokenized text inputs for the language model. These text tokens are processed by an embedding layer h to obtain text token embeddings as {e i } M i=1 ∈ R d . Because we are interested in studying the outputs of specific layers in response to specific tokens, we use g ϕ (.) k,ℓ to denote the output layer embedding corresponding to the k th token position and the layer ℓ. We denote the output of a MLP layer as g ϕ (.) k,ℓ mlp and the output of a self-attention block as g ϕ (.) k,ℓattn .
Causal tracing, derived from the causality literature [28], can be combined with a constraint-based framework to gain causal insights on how information propagates through a model with respect to
Text Constraint Tokens Text Constraint Tokens Other Tokens Other Tokens Last Tokens Last Tokens Restoring MLPs Restoring Self-Attention Early-layers Mid-layers Early-layers Mid-layers Layers Layers specific constraint tokens. This has been used to identify where information is stored in LLMs [23,24] and text-to-image generative models [4]. The central idea of causal tracing is to corrupt a clean model by perturbing the input prompt. The activations of a small subset of layers are then iteratively copied from the clean model to the corrupted model, until the corrupted model restores its output probability to match the clean model's. In this way, we can identify which layers are used to retrieve information relevant to the constraints in the prompt (i.e. causally relate the input to the output).
In LLMs, the model is corrupted by adding a small amount of Gaussian noise to the embeddings of the textual constraint tokens (usually less than 5) [23]. In MLLMs, however, noise must be added to a much larger number of token embeddings -those of the visual tokens (e.g., 576 in LLaVa and 729 in multi-modal Phi-2) from the projection head, and the textual tokens of the visual constraint (e.g. "this director"). Our experiments show that this large noise injection makes it difficult to revert the MLLM to a clean state and recover relevant causal traces (see Fig. 7).
We, therefore, introduce MULTIMODALCAUSALTRACE (see Fig. 2) to address this. Rather than adding Gaussian noise to the embeddings, we instead corrupt the visual constraint token IDs by replacing them with token IDs from a separate word or phrase, such that the visual information is ignored. We illustrate this through an example. (1) Clean Model: Given an image x of the Space Needle, and the question y, "Which city is this building located in?", we compute the probability of the model's output O (e.g., "Seattle") as P clean (O). (2) Corrupted Model: We substitute the visual constraint with an alternative such that the question does not require information from the image to be answered (e.g., "this building" is replaced with "Taj Mahal"). For a multi-constraint question which also has a textual constraint, we can either add Gaussian noise to the textual constraint tokens' embeddings (since there are only a few) or we can similarly replace the token IDs. After all replacements, we ensure that the question still makes semantic sense. We then measure the probability of original output O as P corr (O). With the right corruption, P corr (O) is expected to be low. (3) Restored Model: We then iteratively copy layer activations g ϕ (.) k,ℓ mlp and g ϕ (.) k,ℓattn ∀k ∈ [1, M + N ] from the clean model to the corrupted model, for each layer in turn. For a given layer ℓ and token position k, we denote the restored probability as P restored (O) k,ℓ . After a layer is copied, we observe if P restored (O) k,ℓ is high -indicating that layer ℓ has a strong causal association to the output O. In some cases, no layers have a causal association. We note that this copy operation can be performed over a window of layers {ℓ i } W i=1 at a time, where W is the window size. A window size of 1 copies only one layer at a time. Similar to [23], we track the indirect estimation effect for a layer ℓ as P restored (O) k,ℓ -P corr (O) and use it as a metric to track causal states. Intuitively, this measures the difference in the probability of O under the corrupted model and when a layer ℓ is restored to its original clean state. A high value indicates that the copied layer/s can restore the model to its original clean state (i.e. the layer is causal).
MULTIMODALCAUSALTRACE enables us to identify the specific layers a model retrieves information from in order to answering a visual question. A second component of understanding how MLLMs process factual information is understanding how input prompts are propagated through these storage locations to the model's final output. For this, we use attention contributions [8,37] which compute how much one set of input tokens influences a set of output tokens during the self-attention operation. Specifically, we use this to track (i) how information is transferred from the visual tokens to the causal layers, and (ii) from these layers to the final token, where the final probabilities are computed.
Defining Attention Contributions. The attention operation in a Transformer [32] consists of the query, value, key and output weight matrices:
, where d is the dimension of the internal token embeddings and d h is the dimensionality of the token embedding for a particular attention head h. We define the attention contribution from a token j to token i in layer ℓ as follows:
where the attention matrix for layer ℓ and head h is defined as:
where g ϕ (.) 1:(M +N ),ℓ-1 ∈ R (M +N )×d and A ℓ,h ∈ R (M +N )×(M +N ) . For understanding information transfer from the visual tokens to the causal layers, we use j ∈ [1, M ] and i = c constraint , where c constraint corresponds to the last token in the visual constraint. For understanding the information transfer to the last token, we set j = c constraint and i = c last , where c constraint corresponds to visual constraint's last token and c last corresponds to the last token in the question.
Alongside the above tools, we also introduce a new test-bed called VQA-Constraints to enable our analyses. The test-bed consists of 9.7K natural images paired with factual questions, where each question is annotated with visual and textual constraints (see Sec. 3.1). Specifically, we source image-question pairs from the following datasets i) OK-VQA [21] which covers general knowledge questions, ii) Movies [37] which includes questions about movie directors and awards from Wikipedia, and iii) Known [23] which covers questions about countries, famous people, and places. The questions from the Movies and Known datasets are not originally multi-modal, so we modify them to refer to images which we source from Bing. We provide more details on the dataset construction and the constraint annotation in the Appendix B.
We leverage GPT-4 [27] to annotate the textual and visual constraints in the visual questions in VQA-Constraints. We first manually annotate 100 examples from each dataset with constraints. We provide this as context to GPT-4 and prompt it (see Appendix B) to annotate the constraints in new questions from Multimodal Known and OK-VQA. Due to the templated prompts in Multimodal Movies (e.g., "Name a movie directed by this director? "), the constraints are constant ("this director") across all examples and does not require external annotations.
Using the tools presented in Sec. 3.1, we present our key findings in how MLLMs retrieve information from internal layers and how this information is transferred across the model.
We find that information required to answer a visual question is mainly retrieved from the early-layer MLP and self-attention blocks of a MLLM. This is confirmed by the high indirect estimation effects which MULTIMODALCAUSALTRACE assigns to the early layers in both LLaVa (see Fig. 3, Fig. 8) and multi-modal Phi-2 (see Appendix E) across the three datasets in VQA-Constraints.
This contrasts earlier results for LLMs which have been shown to retrieve information from mid-layer MLPs to answer factual questions [23,24]. To obtain a fairer comparison, we apply MULTIMODAL-CAUSALTRACE to LLaMA and LLaVa which use the same language backbone. We run both on the same set of questions -LLaMA on the Known dataset [23], and LLaVa on our multi-modal version
Dataset : Multi-Known Dataset : Movies Visual Token Index Early Attention Layers Early Attention Layers Late visual tokens Late visual tokens Dataset : OKVQA Late visual tokens Visual Token Index Visual Token Index Early Attention Layers of Known which modifies the questions to refer to an image (see Sec. 3.4). In Fig. 1, we show that the MLPs in the first 4 layers are causal for LLaVa while the MLPs in layers 4-7 are causal for LLaMa.
We also find that causal traces can be extracted from LLaVA with a minimum window size of 1, while LLaMa requires a minimum window size of 5 to obtain any significant causal traces.
Although we find a smaller window size of 1 to provide relevant causal traces (see Fig. 8), we find that a slightly larger window size of 3 results in consistent causal traces across all the questions. In Fig. 3, we report the results using a window size of 3, where we find that the early MLPs as well as self-attention layers are used to retrieve relevant knowledge to answer a visual question. We note that this observation is consistent for all the datasets in VQA-Constraints.
For multi-constraint questions that consist of a visual and textual constraint, information corresponding to the textual constraint is retrieved from a broader set of layers -both the early and mid-layer MLP and self-attention blocks (see Fig. 4). We also find that a larger window size (at least of 6) is required to obtain any causal traces. This suggests that more parametric memory is required to meet both a visual and textual constraint in a given question.
Only a subset of visual tokens are involved in transferring information from the image to the early causal MLP layers.
We also find that the late visual tokens transfer information from the image to the early causal MLP layers (where information is mainly stored) via the first self-attention layer. We show this in Fig. 5 by visualizing the attention values in the early self-attention layers between each visual tokens and the final token in the visual constraint using the method described in Sec. 3.3. We see that the values are the highest in LLaVa's first self-attention layer (which occurs just before the first causal MLP layer), specifically for the last subset of visual tokens (indexes 540-576 out of 576 in total). We hypothesize that these tokens may be summarizing image information that is relevant to the given question before it is transferred and then stored in the MLPs, however we leave this to future study. We note that this pattern holds across all three datasets from VQA-Constraints.
Rather counter-intuitively, we find that even though information is stored in the early layers, the self-attention blocks in the middle layers (rather than layers immediately after) are responsible for propagating this information to the question's final token. The model's answer is sampled at this point, hence the information present here likely influences the ultimate generation. We see this in Fig. 15 and Fig. 16, which plots the attention contribution values between the last visual constraint token and the last token in the question across all the layers, using the methodology in Sec. 3.3. For LLaVa, the self-attention blocks in layers 16-17 are most active. This behaviour is similar in LLMs which also use mid-layer self-attention blocks to transfer information from (mid-layer) stored locations to the last token position.
Mid-layer self-attention contributions can be used to predict whether a MLLM will generate a correct answer, but model confidence is a more reliable predictor When a MLLM generates a correct answer, we observe that the self-attention contributions in its middle layers are higher to when it generates an incorrect answer (see Fig. 15). For LLaVa, this is specifically the attention contributions between the last constraint token and the last prompt to-
Introducing Long-Tailed Knowledge Fixing Error Cases Editing Efficacy Generalization Specificity Editing Efficacy Generalization Specificity 0.2 0.4 0.6 0.8 0.2 0.4 0.6 0.8 Metric Metric ken in the 16th and 17th layer. This holds potential to detect when a model will answer correctly without running a full inference pass, therefore enabling "early" failure mode detection. To investigate this, we compute the scalar average of the attention contributions from these two layers as and find that it can classify a correctly generated answer with an AUROC of 0.63 on the Known dataset in VQA-Constraints (see Fig. 16). We find, however, that the model's confidence -the probability of the generated output token at the final layer -is a slightly stronger predictor, with an AUROC of 0.76 (see Fig. 17). This parallel's previous work in LLMs [37] which used the attention contributions from all the LLM's layers (via a linear model) to predict the generated answer's correctness. In comparison, our results suggest that a subset of MLLM's middle layers alone can be leveraged as a coarse "early" failure mode detector.
Takeaway. MLLMs behave differently in terms of information retrieval from their parametric memory however quite similarly to LLMs in terms of information transfer to the final token.
Previous works have shown that counterfactual information can be inserted into LLMs by editing their causal layers [23,24]. In this section, we verify if a similar approach can be used to edit a MLLM. Specifically, we introduce MULTEDIT, which applies a closed-form update to the early causal MLP layers we identified in Sec. 4. We show that our approach can effectively both (i) fix erroneous answers, and (ii) insert new long-tailed information in LLaVa in a VQA task.
Given an image-question (x, y), we denote a MLLM's generated answer as O. MULTEDIT updates a few parameters in the model such that it generates a new answer O * . Similar to [23,13], we update the W ℓ proj matrix at specific causal MLP layers such that the probability P(O * ) increases. Specifically, we view W ℓ proj as a linear-associated memory (where the matrix's input are treated as keys and its output as values) and uses a closed-form update to map the original keys to new (correct) values. Below, we outline the process for acquiring both the keys and values.
Obtaining keys. Let {v i } N i=1 be the visual token embeddings and {e i } M i=1 the text token embeddings. Given a causal layer ℓ and the last token of a constraint c, we refer to the input of the layer's W ℓ proj matrix as its keys. Specifically, we define the key k c,ℓ to be the input embedding to the W ℓ proj matrix corresponding to the last token of the constraint. This can be obtained with a simple forward pass with the visual and text token embeddings.
Obtaining values. Given a causal layer ℓ, we refer to the output of the layer's W ℓ proj matrix as its values. Specifically, we define z c,ℓ as the output embedding corresponding to the last token of the constraint. We optimize z c,ℓ such that the probability of the correct answer P(O * ) increases as:
where L(z c,ℓ ) is the standard next-token prediction loss used to train LLMs:
MULTEDIT modifies the W ℓ proj matrix such that the old keys k c,ℓ are mapped to the new optimized values z * c,ℓ which increase P(O * ). We define MULTEDIT's editing objective as:
The second regularization term ensures that W ℓ ′ proj , the weights before the edit, do not deviate too much from W ℓ proj . This helps to preserve performance on unrelated VQA pairs.
We experimentally validate MULTEDIT in two real-world model-editing applications:
Fixing incorrect answers to common questions. We test MULTEDIT on a set of ∼450 visual questions which LLaVa answers incorrectly (detected using incorrect VQA accuracy) from the multi-modal Known dataset in VQA-Constraints. These are generally questions about well-known places, persons and brands and companies.
Inserting long-tailed VQA knowledge. We test MULTEDIT on a set of visual questions from the Encyclopedia-VQA dataset [25]. These query fine-grained knowledge about rare landmarks around the world, which MLLMs have been shown to struggle on [25].
For both settings, we compare MULTEDIT to the fine-tuning baselines:(i) fine-tuning from [23] which fine-tunes all the layers using the language modeling objective, and (ii) fine-tuning with constraints from [38] which fine-tunes the layers in a language model with a constraint on the weights to ensure local loss continuity.
We measure the success of each edit operation using the following metrics: (i) Editing Efficacy, which uses P(O * ) to measure the edited model's ability to generate the correct answer for imagequestion (x, y). (ii) Generalization, which uses P(O * ) to measure the edited model's ability to generate the correct answer for the question y paraphrased using a language model (see Appendix for details), and (iii) Specificity, which uses VQA accuracy [1] to measure the edited model's performance on unrelated VQA questions. We consider unrelated questions to be those from the OK-VQA and Movies datasets in VQA-Constraints (see further details in Appendix G).
Overall, our results show that updating the projection matrix using MULTEDIT at just a single early (causal) MLP layer can be a very effective approach for both correcting incorrect answers and inserting new knowledge in MLLMs.
Fixing incorrect answers. In Fig. 6 (left), we show that MULTEDIT is able to successfully fix the generations for all questions with wrong answers. In editing efficacy, the average probability of the correct answer improves from 0.07 to 0.82 after the edit. We also observe strong generalization, with a probability of 0.76 for the correct answer even when the question is paraphrased. Although we see a small drop of 1.5% accuracy on unrelated image-question pairs, we note that MULTEDIT outperforms fine-tuning with and without constraints on all the metrics.
Inserting long-tailed information. In Fig. 6 (right), we show that MULTEDIT is able to reliably insert new long-tailed knowledge in the model with an editing efficacy of 0.83. We see similar strong generalization when paraphrasing questions with an efficacy of 0.79. Similar to above, MULTEDIT incurs a small drop of 1.4% on unrelated questions but is less affected than other methods.
In Fig. 14, we also provide ablations showing that editing the early causal MLP layers leads to better editing efficacies than editing the middle or the later MLP layers.
Our paper takes a closer look at how MLLMs process multi-modal information. We contribute a novel multi-modal causing tracing methodology and test-bed, VQA-Constraints, as well as a range of novel insights on how MLLMs retrieve and transfer information. We also introduce a novel model-editing algorithm, MULTEDIT, which can effectively fix errors or introduce long-tailed knowledge in MLLMs using a simple closed-form update which targets the early causal MLPs. Overall, this work deepens our scientific understanding of recent MLLM architectures, and enables future work in this direction.
The findings of our work are limited to factual questions on natural images with short-form answers. Future work should investigate other types of questions (e.g. subjective), image domains (e.g. charts), and longer-form answers. Our work also does not map visual tokens to specific concepts in the input image -another important research direction. Our editing method, MULTEDIT, while effective for correcting and adding new information, could also be used to add false or harmful information into a MLLM. Work is needed on how to reliably detect misinformation in multi-modal settings.
In Sec. 3.1, we introduce the constraint framework for the task of factual VQA. In particular, there are two types of constraints: (i) Visual Constraint: A set of words in a question which refer to an entity in an image; (ii) Text Constraint: A set of words in a question which together with the visual constraint are responsible for answering a question. In our VQA-Constraints dataset there are two types of questions: (i) Single Constraint Questions: These questions comprise of only the visual constraint; (ii) Multi-Constraint Questions: These questions comprise of both the visual as well as the text constraint. In our paper, we primarily focus on the Single Constraint questions, therefore a majority of questions in our test bed of VQA-Constraints consist of only the visual constraints. Our VQA dataset VQA-Constraints which is annotated with constraints comprise of the following three parts:
OK-VQA: We annotate the questions in the test-set of OK-VQA with the constraints. In total, this part consists of 5k questions. We use the same images as those used in the original OK-VQA test-set.
Multimodal Movies: We use the text-only WikiMovies dataset from [37] and download images using the Bing API using the name of the directors. Our team filters and validates that the downloaded images are of the director itself and there's no noise in the downloaded images. We use a set of 1.5k questions in the final dataset.
Multimodal Known: For understanding knowledge storage in language models, [23] use a probe dataset (known.json) consisting of 1.2k questions. We first replace the subject in the question with a constraint and then download images from Bing API. Our team validates that the downloaded images are correct and uses it to create the Multimodal Known dataset.
Multi-Constraint Questions: The multi-constraint questions in VQA-Constraints are created from OK-VQA and Multimodal Movies. In particular, there are 150 multi-constraint questions for OK-VQA and 500 multi-constraint questions from Multimodal Movies. Given that the primary focus of our paper is on Single Constraint questions, this set of Multi-Constraint questions is relatively small. In total, VQA-Constraints consist of 9.7K VQA questions with Single Constraints and 650 Multi-Constraint VQA questions which we use for interpretability analysis. Annotating and Validating the Visual Constraints. To automatically annotate the constraints in the visual questions from VQA-Constraints, we use a strong language model such as GPT-4. In particular, we annotate only the Multimodal Known and OK-VQA sub-parts of the VQA-Constraints dataset. First from each dataset, we annotate 100 examples which we use as in-context examples to the language model to annotate new questions. Given these annotations, our team then verified if the annotation is correct and if incorrect, modified the constraint annotation to make it correct. In total, the visual constraints are annotated for 9.7k VQA questions.
is better than the middle or the later layers. However, we find that editing the middle or the later layers also lead to a non-trivial editing accuracy, though it's lesser in terms of effectiveness than editing the early causal layers. Recent work [10] also observe this behaviour of non-trivial editing accuracy at locations which are not causal layers.
H Information Transfer and Early Failure Mode Detection -Qualitative Plots
In this section, we first show qualitative in Fig. 15 that the attention contributions from the constraint tokens to the last token is significantly more for correct answers than for incorrect answers. This is quantitatively highlighted and shown in Fig. 16 where we find that Layer 16 and Layer 17 to be the main orchestrator of this difference. A recent work [37] use attention contributions as a signal to detect hallucinations in language models. They show that the attention contribution values when used together with a linear model can be used to predict the correctness of the answer. In fact, they find this metric to be close to the confidence metric (which is the probability of the generated token).
However, [37] necessitates the use of all self-attention layers which reduces it to be used as an "early" failure mode detection metric. For multimodal language models, instead of using all the self-attention layers, we use the average attention contributions from Layer 16 and Layer 17 as an "early" failure mode detection metric. From Fig. 17, we find that the average attention contributions indeed lead to a non-trivial AUROC of 0.63, but it lags behind the confidence metric which obtains AUROC of 0.76. Our early results show that using the model internals can be used to "early" detect if the model is going towards a failure mode, but it lags behind the confidence although that requires the full forward pass.
Layers Layers Correct Answers
Taken from the Bunny repository.