<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>A High-Quality Text-Rich Image Instruction Tuning Dataset via Hybrid Instruction Generation</title></titleStmt>
			<publicationStmt>
				<publisher>Proceedings of the 31st International Conference on Computational Linguistics</publisher>
				<date>01/19/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10637263</idno>
					<idno type="doi"></idno>
					
					<author>Shijie Zhou</author><author>Ruiyi Zhang</author><author>Yufan Zhou</author><author>Changyou Chen</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Large multimodal models still struggle with text-rich images because of inadequate training data. Self-Instruct provides an annotation-free way for generating instruction data, but its quality is poor, as multimodal alignment remains a hurdle even for the largest models. In this work, we propose LLaVAR-2 1 , to enhance multimodal alignment for text-rich images through hybrid instruction generation between human annotators and large language models. Specifically, it involves detailed image captions from human annotators, followed by the use of these annotations in tailored text prompts for GPT-4o to curate a dataset. It also implements several mechanisms to filter out low-quality data, and the resulting dataset comprises 424k highquality pairs of instructions. Empirical results show that models fine-tuned on this dataset exhibit impressive enhancements over those trained with self-instruct data.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Instruction tuning is widely used to improve the generalization and controllability of large language models (LLMs) <ref type="bibr">(Wang et al., 2022;</ref><ref type="bibr">Ouyang et al., 2022;</ref><ref type="bibr">Zhang et al., 2023b</ref>) by converting unseen tasks into instruction-output pairs. Adopting such a manner, visual instruction fine-tuning <ref type="bibr">(Liu et al., 2023a)</ref> enables the visual reasoning capacity of multimodal large language models (MLLMs) by injecting aligned visual tokens leveraging visual encoders such as CLIP-ViT <ref type="bibr">(Alexey, 2020;</ref><ref type="bibr">Radford et al., 2021)</ref> and DINO <ref type="bibr">(Caron et al., 2021;</ref><ref type="bibr">Tong et al., 2024b)</ref>. However, obstacles persist in effectively handling text-centric visual tasks for MLLMs. These challenges likely arise from the underrepresentation of text-rich images in the training dataset, such as COCO <ref type="bibr">(Lin et al., 2014)</ref> and Conceptual Captions <ref type="bibr">(Changpinyo et al., 2021)</ref>. However, the ability to comprehend texts within images is essential for real-world applications. Classical datasets on text-rich images are dedicated to information extraction, such as TextVQA <ref type="bibr">(Singh et al., 2019)</ref>, TextOCR <ref type="bibr">(Singh et al., 2021)</ref>, DocVQA <ref type="bibr">(Mathew et al., 2021)</ref>, and OCR-VQA <ref type="bibr">(Mishra et al., 2019)</ref>. Recent instruction tuning datasets for text-rich images focus more on classical document images, such as Figure <ref type="bibr">(Kahou et al., 2017)</ref>, Chart <ref type="bibr">(Masry et al., 2022)</ref> , and infographics <ref type="bibr">(Mathew et al., 2022)</ref>.</p><p>To address this issue, LLaVAR <ref type="bibr">(Zhang et al., 2023c)</ref> introduces noisy and GPT-4-based instruction-following data of text-rich images, utilizing OCR and caption tools to enhance textual comprehension ability. TRINS <ref type="bibr">(Zhang et al., 2024)</ref> creates a text-rich image instruction dataset in a semi-automatic manner with manual annotation effects to guarantee high-quality captions. In this work, we present LLaVAR-2, a text-rich image instruction-following dataset via hybrid instruction generation between human annotations in TRINS and LLMs to improve the effectiveness of visual instruction tuning. Specifically, we enrich the collected manual captions and the QA dataset with fine-grained details and supplementary self-explain instruction data.</p><p>LLaVAR-2 consists of two parts: LLaVAR-2-Cap for global descriptive captioning on images and LLaVAR-2-VQA for visual question answering. For LLaVAR-2-Cap data collection, rather than directly applying the human-annotated captions from TRINS as answers for crafted instructions, we rewrite the caption by incorporating necessary text/visual details to get detail-enriched captions. Based on it, we construct LLaVAR-2-Cap for precise summarizing to facilitate global visual text understanding. For instruction tuning data, we introduce a novel approach that combines extractive question answering with supplementary rounds of self-explain conversations. These pairs of self-explanations serve to illuminate the rationale behind extractive answers, supported by detailed examinations of relevant image contents. Data with extra pairs of self-explaining bolsters the localized comprehension of text-rich images, offering deeper insights and clearer connections within the visual data. Leveraging the initial high-quality humanannotated captions, VQA and captioning data of LLaVAR-2 are generated via GPT-4o. Compared with TRINS <ref type="bibr">(Zhang et al., 2024)</ref>, LLaVAR-2 shows better diversity in task categories, the length of instructions, and answers. To filter out low-quality data samples, we propose an automatic filtering mechanism for the multimodal instruction tuning dataset, named multimodal Instruction-following Difficulty (mIFD) score and Fact-Following Difficulty (FFD) score, to filter out incompatible extraction and self-explain pairs in VQA data. Our contributions are as follows.</p><p>&#8226; We present LLaVAR-2, a novel dataset consisting of 42k detail-enriched captions and 382k visual question-answering data pairs, all generated automatically using GPT-4o based on human-annotated text-rich image captions.</p><p>&#8226; We design mIFD and FFD scores for filtering on LLaVAR-2-VQA that systematically removes irrelevant or redundant data, ensuring the dataset's high quality.</p><p>&#8226; Beyond demonstrating the dataset's diversity through statistical visualization, we show the superiority of LLaVAR-2 by fine-tuning various base models, showing great improvement on various benchmarks.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Related Work</head><p>Multimodal Large Language Models Recent significant success in multimodal large language models can be traced back to Flamingo <ref type="bibr">(Alayrac et al., 2022)</ref>, InstructBLIP <ref type="bibr">(Dai et al., 2023)</ref> , and LLaVA <ref type="bibr">(Liu et al., 2023b)</ref>. While Flamingo and InstructBLIP form the bridge between language and vision via cross-modal attention, LLaVA-style methods transform visual representations from a standalone visual encoder into visual tokens that language models can understand. Some latest examples include Eagle <ref type="bibr">(Shi et al., 2024)</ref>, Cambrian-1 <ref type="bibr">(Tong et al., 2024a)</ref>, LLaVA-HR <ref type="bibr">(Luo et al., 2024)</ref>, InternVL2 <ref type="bibr">(Chen et al., 2023b</ref><ref type="bibr">, 2024b</ref><ref type="bibr">), Idefics2/3 (Lauren&#231;on et al., 2024b,a)</ref>, LLaVA-OneVision <ref type="bibr">(Li et al., 2024a)</ref> and Qwen2-VL <ref type="bibr">(Wang et al., 2024)</ref>. Scaling up the pre-training and SFT data is a crucial part of improving MLLMs. <ref type="bibr">MiniGPT-4 (Zhu et al., 2023)</ref> uses ChatGPT to produce data compliant with high-quality instructions, while LLaVA <ref type="bibr">(Liu et al., 2023b)</ref> prompts GPT-4 with image captions and boxes to similar ends. Other initiatives <ref type="bibr">(Chen et al., 2023a</ref><ref type="bibr">(Chen et al., , 2024a) )</ref> prompt GPT-4V to generate more than 1 million pieces of quality data for the training of MLLMs. LLaMA-Adapter <ref type="bibr">(Zhang et al., 2023a;</ref><ref type="bibr">Gao et al., 2023)</ref> synchronizes text and image features using COCO dataset inputs. InstructBLIP <ref type="bibr">(Dai et al., 2023)</ref> has restructured 13 vision-language tasks to fit an instruction-based approach. mPLUG-Owl <ref type="bibr">(Ye et al., 2023a,c)</ref> implements multi-task instruction fine-tuning with pre-existing document datasets. VILA <ref type="bibr">(Lin et al., 2024)</ref> equips extra interleaved visual language corpus into MLLMs' pretraining. Cambrian-1 <ref type="bibr">(Tong et al., 2024a) and</ref><ref type="bibr">Idefics2 (Lauren&#231;on et al., 2024b)</ref> create large pools of instruction-tuning data containing a diverse range of visual-language tasks, such as counting, captioning, document understanding, etc. Further research <ref type="bibr">(Liu et al., 2023a</ref><ref type="bibr">(Liu et al., , 2024a;;</ref><ref type="bibr">Bai et al., 2023;</ref><ref type="bibr">Dong et al., 2024;</ref><ref type="bibr">Xu et al., 2024;</ref><ref type="bibr">Luo et al., 2024)</ref> explores enhancing encoder resolution, leading to significant advancements in a variety of downstream applications. For a detailed examination, a comprehensive survey is provided <ref type="bibr">(Li et al., 2023a)</ref>. Despite these advances, visual-text comprehension remains a challenge for many models <ref type="bibr">(Liu et al., 2023c)</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Instruction-Following Data Generation</head><p>To address the lack of sufficient instruction-following data, data synthesis is commonly employed for the training of LLMs/MLLMs. Recent studies have focused on producing high-quality synthetic instruction-following data by leveraging strong and robust LLMs <ref type="bibr">(Wang et al., 2022;</ref><ref type="bibr">Xu et al., 2023;</ref><ref type="bibr">Chen et al., 2023a;</ref><ref type="bibr">Zhang et al., 2023c;</ref><ref type="bibr">Zhao et al., 2023;</ref><ref type="bibr">Zhang et al., 2024)</ref>. For instance, Self-Instruct <ref type="bibr">(Wang et al., 2022)</ref> boosts LLMs' instruction-following capabilities by allowing them to generate their instructional data. WizardLM <ref type="bibr">(Xu et al., 2023)</ref> developed Evol-Instruct to create instruction data with varying levels of complexity. In ShareGPT4V <ref type="bibr">(Chen et al., 2023a)</ref>, the 1.2M image captioning data is expanded by a superb caption model trained on the initial subset. LLaVAR  <ref type="bibr">(Zhang et al., 2023c)</ref> targeted instruction-following data generation for text-rich images by additionally prompting LLMs with high-quality demonstrations. TRINS <ref type="bibr">(Zhang et al., 2024)</ref> further improved upon LLaVAR by leveraging manually crafted highquality image captions instead of BLIP-2 generated ones to generate instructions. Our proposed dataset LLaVAR-2 enriches necessary text and visual details into instruction-following data and maintains the data quality via proposed filtering scores, thus improving visual instruction tuning.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Hybrid Instruction Generation</head><p>In Figure <ref type="figure">1</ref>, we show the data collection pipeline of Hybrid Instruction including the detail-enriched captions LLaVAR-2-Cap in Section 3.1 and the visual question answering data LLaVAR-2-VQA collection in Section 3.2. We conduct data filtering for LLaVAR-2-VQA, with the filtering process described in Section 3.3. Examples of LLaVAR-2-Cap and LLaVAR-2-VQA are given in Appendix C and D respectively.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.1">Detail-enriched Captions</head><p>Manual captions in TRINS provide concrete descriptions of visual components but often offer only general and succinct summaries for text-heavy areas, lacking explicit text labels for these summaries. Thus, clear links between descriptions and corresponding text or image regions are hard to construct for MLLMs. To address this, based on OCR results, we leverage advanced GPT-4o to enrich these captions with missing details and text labels.</p><p>Specifically, to enhance global awareness of precise geometric relationships in text-rich images, we combine the text-rich image, OCR results with bounding boxes and manual caption automatically by prompting them to GPT-4o and asking for captions with more text labels while preserving the original style and structure of its manual caption. The system message used is detailed in Appendix B.</p><p>Furthermore, although the manual results may contain errors, OCR bounding boxes help GPT-4o better locate text and correct inaccuracies as shown in the example of Figure <ref type="figure">1</ref>. These detailenriched captions are further paired with descriptive questions to create single-round caption data D g :{Q g , A g }, named LLaVAR-2-Cap, which includes detailed visual text and object descriptions.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Visual Question Answering with Self-explanations</head><p>Visual question-answering data is crucial for visual instruction tuning and typically includes extractive QA data that focuses on the local attributes of an image I. In LLaVAR-2-VQA, each extractive QA pair D e :{Q e , A e } is accompanied by a self-explain pair D r :{Q r , A r } to illuminate the rationale behind the extractive answer A e . The detailed data collection The background shows a view of a beautiful city with buildings, a bridge over the sea, and a person walking past it.</p><p>Self-explain Q1: What visual elements in the background contribute to the description of the cityscape on the magazine cover? Self-explain A1: The background includes a view of a city with buildings and a bridge over a body of water, along with a person walking past it. These elements collectively create a picturesque representation of a vibrant urban landscape, contributing to the description of a beautiful city.</p><p>Extractive Q2: What could be the focus of the article "MIDNIGHT CALLING"? Extractive A2: The article "MIDNIGHT CALLING" seems to focus on the nightlife and after-dark hotspots in Dublin. It likely highlights the best places to visit, such as bars, clubs, late-night eateries, and other entertainment venues that come alive at night. This article would appeal to readers interested in exploring the city's vibrant night scene.</p><p>Self-explain Q2: How does the article title "MIDNIGHT CALLING" and its positioning on the cover suggest its content? Self-explain A2: The article title "MIDNIGHT CALLING" is positioned in the middle left of the cover in bold yellow text, indicating its importance. The title itself suggests content related to nightlife and after-dark activities, implying that the article will highlight hotspots and entertainment venues that are popular during the night. pipeline for D e and D r is described following.</p><p>Extractive Question Answering Data In addition to these globally descriptive instructionfollowing data, instructions focusing on specific local attributes of text-rich images are also necessary. Thanks to the high-quality human-annotated captions, which clearly indicate each attribute of both text and visual areas in the image, it is convenient to utilize GPT-4o to generate conversational data that focuses on partial attributes in a local area. To achieve this, we prompt GPT-4o with the text-rich image, its human-annotated caption, OCR results from PaddleOCR, and two in-context demonstrations following <ref type="bibr">(Zhang et al., 2023c</ref>).</p><p>Applying the system message in <ref type="bibr">(Liu et al., 2023b)</ref>, this setup generates multiple single-turn conversations D e :{Q e , A e } around an image, each focusing on different components without overlapping. For each single-turn conversation, we formulate the question and answer as the input instruction and the target response, respectively.</p><p>Self-explain Conversational Data While the extractive instruction-following data D e focusing on local attributes will facilitate the understanding of text-rich images, it still presents challenges for MLLMs to answer extractive questions when the model needs to link a visual area with a text area or when the question involves one text area but the answer is found in another. In these cases, precise associative and implicit reasoning is necessary. Although D e provides accurate retrieval labels, it lacks clarity on how its extraction is achieved. The answering process behind D e can supplement the extract result in A e with text/geometric labels, such as clear indications of where the query object is located in the image, thus improving MLLMs for better recognition and extraction.</p><p>To help MLLMs handle challenging scenarios, we make the implicit extraction process explicit by generating a self-explain conversation D r :{Q r , A r } as the explanation for each pair D e . Prompting GPT-4o with the text-rich image, manual caption, OCR results, corresponding D e , and system messages, the self-explain dialogue D r is generated to explain how the extractive answer D e is obtained, highlighting implicit connections between different components. Please refer to Figure <ref type="figure">1</ref> for an example of the two dialogues. The self-explain dialogue D r that is based on D e will enhance MLLMs' ability to recognize implicit connections in images, improving performance in tasks such as extractive QA and text-rich image analysis, while avoiding verbosity if directly adding reasoning into extractive answers. The system message used for generating D r is detailed in Appendix B.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">Instruction Data Filtering</head><p>Both initial extractive QA D e and self-explain QA D r of LLaVAR-2-VQA generated in Section 3.2 may include low-quality samples. Specifically, the low-quality extractive QA D e generally fails in two cases: (1) the entire D e is barely related to the input image i but focuses on an unfamiliar or too abstract topic; (2) the extractive answer A e is antiintuitive or unrelated to the extractive question A e .</p><p>LLaVAR-2-Caption: This is an advertisement for the 3rd Annual Bob Marley Tribute, presented by Highlanda.net and The Honorary Citizen. The cover features a stylized image of Bob Marley. The main title in the middle of the cover reads "3RD ANNUAL BOB MARLEY TRIBUTE". At the bottom center, in a red box, it mentions the event "WILD PITCH" on February 12th, 2017, located at 255 Trinity Ave, Atlanta, GA. The event timing is from 6 PM to 1 AM, and tickets are available at Highlanda.net for $15 online and $20 at the door. Additional details include an early live reggae mixer from 6-8 PM featuring Ijahknowah and the B.R.A.P Band. The Rub-A-Dub Session from 8 PM to 1 AM will feature Highlanda Sound, Agard, DJ Passport, Black Magic, Natural Vibes, and Innocent Sound. There will be eats by Webba's Jerk Hut.</p><p>Extractive Q1: What type of music and dance style is being celebrated in the event? Extractive A1: The event celebrates a style of Jamaican reggae called "Rub-a-Dub," which emerged in the 70s and 80s. The term Rub-a-Dub comes from a dance style where a man and woman rub up very close together."</p><p>Extractive Q2: When and where is the event taking place? Extractive A2: The event is taking place on February 12th, 2017, at 255 Trinity Ave, Atlanta, GA. Self-explain Q2: What information in the image specifies the date and location of the event? Self-explain A2: The date and location of the event are specified in the red box in the lower center of the image. It states "February 12th, 2017" and "255 Trinity Ave, Atlanta, GA ".</p><p>Extractive Q3: Who is the event a tribute to? Extractive A3: The event is a tribute to Bob Marley. Self-explain Q3: How is it indicated that the event is a tribute to Bob Marley? Self-explain A3: The tribute to Bob Marley is indicated by the large text in the middle of the image that reads "3rd Annual Bob Marley Tribute". While both cases may happen together, we develop an effective filtering strategy based on Instruction Following Difficulty(IFD) <ref type="bibr">(Li et al., 2023c)</ref>. Given the instruction, the corresponding answer, and the model &#969;, IFD is calculated as:</p><p>where s &#969; is the sum of the next token prediction loss of the input computing by the model &#969;. s &#969; (A | Q) inputs model &#969; with {Q, A} and sums the loss of each A's token. s &#969; (A) is computed without the instruction A input. IFD can measure how helpful the instruction is towards the generation of the answer, and smaller IFD reflects better correspondence and compatibility between them. Inspired by IFD, we propose Multimodal Instruction-Following Difficulty (mIFD) to filter D e as:</p><p>In mIFD score, the first term Visual-Following Difficulty (VFD) takes the entire extractive dialogue D e and the image as the input to evaluate the correspondence between them. The second term directly utilizes the IFD score to detect samples in case (2).</p><p>Computing mIFD scores on D e , we excluded the highest 70% of samples. Examples filtered out are shown in Figure <ref type="figure">15</ref>. We can find that many unrelated answers A e are generated based on MLLMs' pre-knowledge of the object.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: Why is the additional information "KEY TAKEAWAYS, SUMMARY &amp; ANALYSIS INCLUDED" significant? A: The additional information "KEY TAKEAWAYS, SUMMARY &amp; ANALYSIS INCLUDED" is significant because it indicates that this edition of the book provides a comprehensive overview, including key points, summaries, and analyses. This makes the content more accessible and understandable for readers.</p><p>[</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>irrelevant and nonspecific contents]</head><p>Case 1 : FFD score = 0.9484 Extractive Q: How does the cover image look? A: The cover image features a small figure approaching a blue sci-fi pyramid with floating rocks around it. There is an ornate sci-fi pattern graphic with three full moons under and around the title.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: What elements are depicted in the cover image? A: The cover image depicts a small figure approaching a blue sci-fi pyramid with floating rocks around it, and there is an ornate sci-fi pattern graphic with three full moons under and around the title.</p><p>[repetitive text]</p><p>Case 2 : FFD score = 0.1796 As for the data filtering of self-explain data D r , we find abnormal D r sometimes falls into two cases: (1) D r focuses on a new concept unrelated to the image and D e ; (2) D r is over-corresponded with D e leading to A r very similar with A e . Examples of 2 cases are shown in Figure <ref type="figure">5</ref>.</p><p>Extending IFD to the dialogue level, we propose the Fact-Following Difficulty (FFD) score to reflect the closeness between the extractive data D e and the self-explain data D r , which is formed as:</p><p>.</p><p>(3)</p><p>From Figure <ref type="figure">5</ref>, high FFD reflects unrelated D r and D e , while low FFD score indicates repetitive answers. Utilizing Phi-3.5-vision to compute FFD, we filter out 1.5k poorly correlated and 5.6k overrelated self-explain pairs D r . Examples of filteredout data are shown in Appendix E.</p><p>Furthermore, we verify the consistency of mIFD and FFD score computing by different MLLMs. Their distributions are shown in Figure <ref type="figure">4</ref>. We can observe that their FFD scores share a very similar FFD pattern. While their IFD scores have slight differences, their mIFD scores computed using the IFD term keep these differences consistent, e.g. IFD computed by LLaVA-1.5 shows a narrower lower tail and this difference remains in mIFD.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Enabling LLaVA to better Read</head><p>To verify the benefits of LLaVAR-2 for visual text understanding, we propose LLaVAR-2-3.8B following the similar architecture of LLaVA <ref type="bibr">(Liu et al., 2023b)</ref>. LLaVAR-2-3.8B is an efficient multimodal language model leveraging the smallscale but effective microsoft/Phi-3-mini <ref type="bibr">(Abdin et al., 2024)</ref> as the language decoder D and adapting the Mixture-of-Resolution Adaptation (MRA) in LLaVA-HR <ref type="bibr">(Luo et al., 2024)</ref> to build the vision encoder V . Specifically, CLIP-ViT-L and CLIP-ConvNeXt-L are utilized to encode lowand high-resolution images. High-resolution features are embedded into low-resolution paths by applying MRA. Integrating high-resolution embeddings in this manner enhances the MLLM's image comprehension ability, especially for text-rich images, while maintaining efficiency without extra visual tokens. The grid embeddings before the last layer of the transformer are aligned to the word embedding space via a two-layer MLP projector.</p><p>We conduct two-stage training following LLaVA-HR that optimizes the projector only without the MRA module in stage 1 and fully optimizes the entire MLLM during stage 2. Besides the 158K instruction-following data from LLaVA, our 40k </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Dataset Analysis</head><p>Besides scaling dataset size, improving data diversity is essential for enhancing end-to-end visual instruction tuning in MLLMs. The variety of instructions across concepts and tasks determines a model's ability to understand and navigate text-rich images with complex semantics. In this section, we evaluate the instruction diversity of LLaVAR-2-VQA to assess its quality.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Statistics and Analysis</head><p>In Figure <ref type="figure">6</ref>, we show the visualization of instruction distribution of LLaVAR-2-VQA and TRINS-VQA <ref type="bibr">(Zhang et al., 2024)</ref>, a high-quality text visual instruction-following dataset, based on questioning words and keywords statistics following <ref type="bibr">(Wang et al., 2022)</ref>. The inner circle illustrates the Abstract and Extract instruction's distribution decided by keywords in questions. The outer circle reflects the distribution of questioning words in LLaVAR-2-VQA. It shows overall more diverse patterns of LLaVAR-2-VQA than TRINS-VQA and the emergence of self-explain data prominently enriches the diversity, e.g. self-explain questions beginning with "Which" and "How" ask for the exact image sources for the extractive answer Q e . In Figure <ref type="figure">7</ref>, LLaVAR-2-VQA (averages 12.4 words per question, 38.9 words per answer) shows a more balanced question and answer length distribution than TRINS-VQA (averages 10.6 words per question, 24.3 words per answer), indicating a greater variety of complexity levels in its QAs. Additional statistics are shown in Appendix A.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Diversity Evaluation via Embedding Distance Measurement</head><p>The statistics and visualization in Embedding Cat. Task2Vec&#8593; S-BERT&#8593; Ours 0.1444 0.6334 TRINS 0.1156 0.5410 Table 1: Instruction Diversity Coefficient of LLaVAR-2-VQA and TRINS-VQA based on Task2Vec and Sentence-BERT Section 5 provide clear sights of how the LLaVAR-2-VQA is diversely composed. However, this evaluation based on single-word extraction using manual or neural parsers is not representative enough to reflect the precise instruction coverage. To study a more accurate diversity evaluation of LLaVAR-2-VQA, we use the intra-dataset diversity coefficient (Lee et al., 2023) to compute the dataset diversity indicator based on the taskspecific TASK2VEC (Achille et al., 2019) embedding and the general sentence-level Sentence-BERT <ref type="bibr">(Reimers and Gurevych, 2019)</ref> embedding. Specifically, we formed the intra-dataset instruction diversity coefficient following <ref type="bibr">(Lee et al., 2023)</ref> as:</p><p>where B 1 and B 2 are batches sampled from the target dataset D and f B i denotes the batch-level embedding which is the diagonal of Fisher Information Matrix for TASK2VEC or the mean of the instructions' Sentence-BERT embeddings of the batch. We apply Cosine distance d to measure the semantic gap between sampled batches within the target dataset. Thus, larger div(D) indicates higher instruction diversity. While TASK2VEC provides fine-grained task-level diversity measurement, Sentence-BERT indicates general semantic diversity.</p><p>We compare the instruction diversity coefficient LLaVAR-2-VQA with TRINS-VQA in Table <ref type="table">1</ref>, with batch size &#8595;B&#8595; = 512 and 200 batches for each dataset. As shown, LLaVAR-2-VQA exhibits a higher diversity level under both embedding settings, indicating the enriched instruction category and semantics discussed in Section 5.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">Experimental Results</head><p>We aim to illustrate the benefits of integrating LLaVAR-2 in visual instruction tuning. LLaVAR-2 models are evaluated on LLaVAR-2-Cap, LLaVAR-2-VQA and classical text-rich benchmarks to demonstrate the effect of LLaVAR-2 to MLLMs. All experiments are implemented with PyTorch and performed on Nvidia A100 GPUs.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.1">Visual Question Answering</head><p>We first evaluate LLaVAR-2 models and baselines LLaVAR-2 in LLaVAR-2-VQA shown in Table 2. We report Extract Accuracy following <ref type="bibr">(Wu et al., 2023)</ref>, text similarity metrics BLEU <ref type="bibr">(Papineni et al., 2002)</ref>, METEOR <ref type="bibr">(Banerjee and Lavie, 2005)</ref>, ROUGE <ref type="bibr">(Lin, 2004), and</ref><ref type="bibr">CIDEr (Vedantam et al., 2015)</ref>. LLaVAR-2 with Phi-3-Mini as the backbone outperforms other methods in zeroshot VQA on LLaVAR-2-VQA. Thanks to the enriched details and the implicit answer process in LLaVAR-2, LLaVAR-2 models can have a better comprehension of visual texts and generate precise extractive answers reflecting from the metric of Extract Accuracy. While sentence similarity metric's results can reflect how well the model can deal with complex extract questions and to what extent the model understands the reasoning process behind the extractive result regarding the self-explain QA set, LLaVAR-2-3.8B's better results on these metrics demonstrate the benefits of LLaVAR-2 on these two perspectives and also indicates the interplay of extractive and self-explain pairs in LLaVAR-2-VQA. Comparing the performance of LLaVAR-2 and LLaVAR-2 w/o D r , the supplement of the self-explain pair D r after D e can enhance the model's capacity toward text-rich images. As for other methods, MiniCPM-V <ref type="bibr">(Yao et al., 2024)</ref>, Cambrian-1 <ref type="bibr">(Tong et al., 2024a)</ref> , and Phi-3/3.5-vision <ref type="bibr">(Abdin et al., 2024)</ref> perform well on LLaVAR-2-VQA, highlighting the importance of including diverse visual text data during training.</p><p>In addition, we also conduct evaluations on the classical visual text understanding benchmarks <ref type="bibr">(Liu et al., 2023b)</ref> and OCR-Bench <ref type="bibr">(Liu et al., 2023c)</ref> shown in Table <ref type="table">3</ref> and <ref type="table">Table 4</ref>. For benchmarks involving images that include both textual and abstract visual elements, such as DocVQA <ref type="bibr">(Mathew et al., 2021)</ref>, ChartQA <ref type="bibr">(Masry et al., 2022)</ref>, and InfoVQA <ref type="bibr">(Mathew et al., 2022)</ref>, LLaVAR-2 models significantly outperform other</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Method</head><p>Extract Acc. B@1 B@2 B@3 B@4 METEOR ROUGE CIDEr Phi-3-vision-128k <ref type="bibr">(Abdin et al., 2024)</ref> 22 Table 3: Zero-shot performance (accuracy %) on text-based VQA. We use &#8224; to refer to the results obtained from previous work (Liu et al., 2023c), and use bold and underline to indicate the best and second best results, respectively. Method Recog. VQA S VQA D KIE Total Gemini 215 174 128 134 651 GPT-4v 167 163 146 160 636 Text-Monkey 169 164 115 116 561 Monkey 174 161 91 88 514 mPLUG-Owl2 153 153 41 19 366 LLaVAR 186 122 25 13 346 LLaVA1.5-13B 176 129 19 7 331 MiniGPT-V2 124 29 4 0 157 LaRA-13B 206 151 101 145 603 LLaVAR-2 (Phi-3-Mini) 225 152 114 143 634 LLaVAR-2 (Vicuna-1.5) 241 162 121 156 680</p><p>Table 4: Results of MLLMs on OCRBench. Recog. represents text recognition, VQA S represents Scene Text-Centric VQA, VQA D represents Document-Oriented VQA. We use bold and underline to indicate the best and second best results, respectively.</p><p>methods, highlighting the importance of high proportion of abstract QAs within LLaVAR-2-VQA to improve abstract document understanding. Additionally, while these three datasets are involved in the training of Monkey <ref type="bibr">(Li et al., 2024b)</ref> and TextMonkey <ref type="bibr">(Liu et al., 2024b)</ref>, LLaVAR-2 per-forms better than these finetuned models on these three datasets, thanks to high-quality text-centric VQAs in LLaVAR-2-VQA/Cap. For scene textcentric ST-VQA <ref type="bibr">(Biten et al., 2019)</ref> and TextVQA <ref type="bibr">(Singh et al., 2019)</ref>  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.2">Text-rich Image Captioning</head><p>We evaluate the captioning capacity of LLaVAR-2 models and baselines on LLaVAR-2-Cap, which requires MLLMs to generate summaries and keep necessary text labels at the same time. We compare the performance on this challenging task in Table <ref type="table">5</ref>.</p><p>We can observe that MiniCPM-V, Cambrian-1, and Phi-3/3.5-vision achieved remarkable improvement upon classic MLLMs such as LLaVA-1.5. It is due to the considerable attention of these models on text-heavy visual tasks, for example, MiniCPM-V <ref type="bibr">(Yao et al., 2024)</ref> includes a large amount of text-rich captioning data in its stage-1 and 2 training, Phi-3.5-vision's post-training is involved with diverse text image tasks. Fine-tuned on LLaVAR-2-Cap, LLaVAR-2 models outperform other methods on all the text similarity metrics, indicating the necessity of the comprehensive captions in LLaVAR-2-Cap. Besides, the Mixture-of Resolution Adaptation used in the visual encoder of LLaVAR-2 models adopted from LLaVA-HR plays a crucial role in making accurate summaries in captioning tasks. Among LLaVAR-2 models with different backbones, LLaVAR-2 (Vicuna-1.5 13B) achieved the best result, while smaller-sized backbones such as Phi-3-Mini and Llama-3.1 8B still obtained comparable performance. LLaVAR-2 (Phi-3-Mini) w/o D g is only fine-tuned on the VQA data without adaptation on captioning data. Its not ideal result on captioning tasks further demonstrates the essential of global descriptive data for summarizing tasks.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.3">Additional Experiments on Data Filtering</head><p>In this section, we aim to verify the effectiveness of the proposed mIFD and FFD scores. We apply the mIFD score to select the extractive data D e first and then filter D r via FFD score. Filtering out data ranging from 10% to 90%, We apply the different filtered LLaVAR-2-VQA train set as the only fine-tuning dataset for different checkpoints to verify the effectiveness of the proposed filtering method. In Figure <ref type="figure">8</ref>, we present the BLEU-1 and CIDEr results for different filtering percentages. It is evident that 70% is the sweet spot for LLaVAR-2-VQA that our filtering scores are efficient before 70% and after 70% performance drops due to the limited size of data for fine-tuning. These results demonstrate that the proposed filtering score can select high-quality samples enabling the model to reach better performance with a smaller data size.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">Conclusions</head><p>We propose Hybrid-Instruct, an automatic multimodal instruction generation framework based on manual captions for visual instruction tuning tailored to text-rich images, to construct LLaVAR-2 data. The detail-enriched captions and self-explain dialogues in LLaVAR-2 enhance the performance of MLLMs on different benchmarks. In addition, the proposed filtering score mIFD and FFD are shown to be effective in filtering out unqualified dialogues in the LLaVAR-2 given the image and extraction contexts. In general, LLaVAR-2 introduces novel approaches to generate and select diverse and high-quality instruction-following data for MLLMs.</p><p>A LLaVAR-2 Statistics Number of images 42870 Number of detail-enriched captions 42870 Number of LLaVAR-2-Cap pairs 42870 Average # of words per detail-enriched captions 114.5 Number of LLaVAR-2-VQA pairs 382406 Average # of words per VQA question 12.4 Average # words per VQA answer 38.9 Table 6: LLaVAR-2 Dataset Statistics B Details for LLaVAR-2 Data Collection System message for the generation of detailenriched captions A g in LLaVAR-2-Cap: You are an AI visual assistant, and you are seeing a single image. What you see is provided with an image, one corresponding OCR result, and one corresponding image caption describing the information within the same image you are looking at. Image captions and OCR results might include hallucinations, while OCR results are more accurate. OCR results are constructed with [[4 boundingbox coordinates], text, OCR confidence]. Enrich the image caption with detailed support text and location information from the OCR result.</p><p>The enriched image caption should be in a tone that a visual AI assistant is seeing the image and describing the image. Maintain the content in the original image caption while adding the support information from the OCR result to its corresponding part in the original image caption for a detailed description:</p><p>(1) for the support text added to the image caption, its bounding box coordinates should indicate the similar location described in the corresponding part of the original image caption;</p><p>(2) DO NOT mention OCR bounding box coordinates in your caption. Provide the location information with the same style as the original image caption;</p><p>(3) DO NOT add information that looks unrelated to or contradicts OCR results;</p><p>(4) When OCR results include new information, enrich this content into the original caption;</p><p>(5) When OCR results and image caption has large difference in describing the same area of the image, maintain the description in the original image caption;</p><p>(6) Do not include garbled characters in the generation.</p><p>System message for the generation of extractive conversation D e in LLaVAR-2-VQA following <ref type="bibr">(Liu et al., 2023b)</ref>:</p><p>You are an AI visual assistant, and you are seeing a single image. What you see is provided with two OCR results and one image caption describing the information within the same image you are looking at. Image captions might include hallucinations, while OCR results are more accurate. Answer all questions with definite answers as you are seeing the image. Design a conversation between you and a person asking about this photo. The answers should be in a tone that a visual AI assistant is seeing the image and answering the question. Ask diverse questions and give corresponding answers.</p><p>Include questions asking about the visual content of the image (e.g., the man, the sunset, the ocean.) and the texts contained in the image. Only include questions that have definite answers:</p><p>(1) one can see the content in the image that the question asks about and can answer confidently;</p><p>(2) one can determine confidently from the image that it is not in the image. Do not ask any questions that cannot be answered confidently;</p><p>(3) DO NOT mention OCR or image caption in your questions and answers;</p><p>(4) DO NOT ask about information from captions while it looks unrelated to or contradicts OCR results. Also include complex questions that are relevant to the content in the image, for example, asking about background knowledge of the texts in the image, asking to discuss the design of the image, etc. Again, do not ask about uncertain details. Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary.</p><p>System message for the generation of selfexplain conversation D r in LLaVAR-2-VQA:</p><p>are an AI visual assistant, and you are seeing a single image. What you see is provided with an image, one corresponding OCR result, one corresponding image caption describing the information within the same image you are looking at, and a set of reference QAs on this image. OCR results contain text description with location information which is constructed with [[4 bounding-box coordinates], text, OCR confidence]. The image caption contains a visual description from the human. A set of reference questions and answers around this image are provided after the image, OCR result, and image caption. Based on them, generate a pair of reasoning QA that asks why/how/where each provided reference answer looks like it and explains it in the generated reasoning answer The generated reasoning QA should be in a tone that a visual AI assistant is seeing the image and answering the question: (1) The generated question can ask the reason behind the provided reference QA's answer or ask how to get the provided QA's answer according to the provided context; (2) If the reasoning process for the reference QA is very explicit, the generated reasoning QA should focus on the source of the reference answer, e.g. for simple extractive question answering, the reasoning QA should focus on where the extraction of the target object of the reference question from in the image and its content in the image; (3) If the reasoning process for the reference QA is complex, the generated reasoning QA should reason on the relationship/connection between the involved objects (text object, visual object if needed) in the image; (4) The generated questions should have diverse styles, do not only use a single style format; (5) Find supportive information from the provided image/caption/OCR results to answer the generated reasoning question, but only assuming you can see the image, do not mention that you can see the caption/OCR results; (6) Make sure the generated reasoning QA corresponds/very related to the reference QA and the image, avoid only extending new concepts in the reasoning QA; (7) Generated reasoning QA should be in the same format as the provided QA:'Reasoning Question: {generated reasoning question}\\n Reasoning Answer: {generated reasoning answer}'. C LLaVAR-2-Cap Examples This is the title page of the book titled \"ALEISTER CROWLEY AND THE AEON OF HORUS.\" The book discusses themes such as History, Magick, Psychedelia, and UFOlogy. The author's name, Paul Weston, is prominently displayed at the bottom center of the page. The cover art features an illustration of hands holding a flaming chalice, with mystical figures and symbols surrounding it. On the left side, there's a figure with a bird's head and a red disk, while on the right side, there's another mystical figure. The background is adorned with cosmic and spiritual imagery, enhancing the esoteric theme of the book. This is a promotional ad celebrating the 2019 Bank of America Chicago Marathon. In the lower center, the bold title \"26.2 Miles of Memories\" is prominently displayed, with the encouraging subtext \"Congratulations, you did it!\" just below. Below this, the logos for the Chicago Tribune and the Chicago Marathon are also visible.\n\nThe ad background mimics a newspaper layout split into two pages. The left page features the headline \"Chicago Tribune 2019 Bank of America Chicago Marathon Commemorative Section\" at the top. Below this, there is a vibrant photo capturing a large crowd of runners beneath a banner that reads \"Bank of America Chicago Marathon\" set against an urban backdrop with tall buildings.\n\nThe right page is titled \"Bank of America Chicago Marathon Results\" and is densely filled with the detailed race results, listing numerous participants' names and times. The overall background of the ad is plain white, enhancing the prominence of the colorful and detailed newspaper sections 10106 This is a movie poster for \"Gentleman's Agreement.\" The poster prominently features the names of the leading actors: Gregory Peck, Dorothy McGuire, and John Garfield, all in large blue text on the right side. The phrase \"DARRYL F. ZANUCK presents\" is displayed at the top. Below the cast names, it is noted that the movie is based on Laura Z. Hobson's work.\n\nThe title \"Gentleman's Agreement\" is displayed prominently in large yellow and red text towards the bottom center of the poster. The poster also includes additional cast names at the bottom: Celeste Holm, Anne Revere, June Havoc, Albert Dekker, Jane Wyatt, and Dean Stockwell. Production credits include Darryl F. Zanuck as the producer, Moss Hart for the screenplay, and Elia Kazan as the director.\n\nIn the background, there is an illustration of a man sitting and appearing pensive, while a woman stands beside him. Additionally, the poster features a \"20th CENTURY-FOX\" logo at the bottom right corner, highlighting the studio behind the film. This is a blog image titled \"Capricorn Rituals.\" The title is prominently displayed in large white and green text at the upper center of the image. Below the title, various rituals are listed, each accompanied by an illustration. The subtext reads:\n\n-\"meditate,\" illustrated with a gesturing hand on the left side of the image.\n-\"write down long-term intentions,\" depicted with a list of intentions in the centerright part of the image.\n-\"eat plant-based for a day, buy a plant, or plant a sapling for your home!\" shown with hands cupping a plant in the center-right section.\n-\"go for a silent walk or hike listening to the sounds of nature,\" illustrated with a girl in an orange jacket walking next to an orange leaf tree at the bottom left.\n-\"get rid of one distraction for a week,\" depicted with a cell phone and likes and heart emojis coming out of it in the bottom right.\n\nEach ritual is visually represented, creating a cohesive and engaging guide for Capricorn rituals. This is a book cover featuring the title \"HELLO, NEIGHBOR!\" in large red text in the upper center. Below the title, in smaller blue and red text, it reads \"The Kind and Caring World of Mister Rogers.\" On the lower right side, black text states, \"by Caldecott Medalist MATTHEW CORDELL.\" An illustrated gold seal on the left side reads \"The Authorized Picture Book Biography.\" The background illustration showcases an elderly man with gray hair and a red sweater standing behind a miniature town with brown buildings and green trees, accompanied by a cat. This is a sound circus cover post. The title reads \"JOANNA MACGREGOR GERSHWIN: Rhapsody in Blue, Piano Concerto in F, The Gershwin Songbook, Broadway Arrangements.\" The background cover image features a lady with blonde hair wearing a black and gray dress, standing in front of old brick buildings. Extractive Question: What is the price for a VIP meet and greet with both Lifehouse and Switchfoot? Extractive Answer: The price for a bundle VIP meet and greet with both Lifehouse and Switchfoot is $135. Self-explain Question: What details are provided in the ad to determine the price of a VIP meet and greet bundle with both Lifehouse and Switchfoot? Self-explain Answer: The price for a VIP meet and greet bundle with both Lifehouse and Switchfoot is listed under \"OPTION 3\" in the VIP Experience Options section. It states \"BUNDLE: LIFEHOUSE &amp; SWITCHFOOT VIP MEET &amp; GREET -$135,\" providing the specific cost. Extractive Question: Who awarded Nilanjan Saha the title of \"The World's Best Brother\"? Extractive Answer: The title of \"The World's Best Brother\" was awarded to Nilanjan Saha by Sayantani. Self-explain Question: What indicates that Sayantani awarded the title to Nilanjan Saha? Self-explain Answer: The award plaque explicitly states \"Awarded by: Sayantani\" at the bottom right corner, confirming that Sayantani is the one who awarded the title to Nilanjan Saha. Extractive Question: What is the main subject of the book \"How to Think Like a Roman Emperor\"? Extractive Answer: The main subject of the book \"How to Think Like a Roman Emperor\" is the Stoic philosophy of Marcus Aurelius. Self-explain Question: What elements on the book cover indicate the main subject of \"How to Think Like a Roman Emperor\"? Self-explain Answer: The main subject is indicated by the subtitle \"The Stoic Philosophy of Marcus Aurelius\" which clearly states the focus on Stoic philosophy through the figure of Marcus Aurelius.</p><p>Extractive Question: How many volumes are there in this series, and which volume is this book?</p><p>Extractive Answer: This book is the first of two volumes in the series.</p><p>Self-explain Question: What information on the cover indicates the number of volumes in the series and the specific volume of this book? Self-explain Answer: The information \"Book 1 of 2\" is located towards the bottom center of the cover, indicating that this is the first of two volumes in the series. Extractive Question: What is the main theme suggested by the short context on the cover? Extractive Answer: The main theme suggested by the short context on the cover is about a character attempting to break through the emotional barriers of someone who is perceived as cold-hearted. It hints at themes of love, perseverance, and possibly redemption, as one character strives to reach the heart of another.</p><p>Self-explain Question: What elements of the short context hint at the main theme of the book? Self-explain Answer: The elements \"cold-hearted bastards\" and \"working her way into his heart\" suggest themes of love, perseverance, and emotional breakthrough, indicating a narrative focused on overcoming emotional barriers and possibly redemption.</p><p>Extractive Question: What organization or publisher logo is visible on the book cover?</p><p>Extractive Answer: The logo visible on the book cover is \"GRIN.\"</p><p>Self-explain Question: Where is the logo of the organization or publisher located on the book cover? Self-explain Answer: The logo \"GRIN\" is located at the bottom right corner of the book cover, as indicated by the small green and white logo. Extractive Q: Can you describe the design of the book cover?</p><p>A: Yes, the book cover has the title \"Cars and How to Drive Them\u2014Primary Source Edition\" written in a large, gold font at the center. The background is dark blue with a textured pattern. Below the title, the word \"Anonymous\" is written in a smaller gold font.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: How does the title \"Cars and How to Drive Them: Primary Source Edition\" indicate the nature of the book's content? A: The title \"Cars and How to Drive Them: Primary Source Edition\" indicates that the book contains information on driving cars, and the label \"Primary Source Edition\" suggests that this information might be original and unaltered, providing firsthand accounts or instructions from the time it was first published. This implies the content could be historically significant and authentic.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>[irrelevant Self-explain]</head><p>FFD score = 1.070361</p><p>Extractive Q: What is the content of the Amharic text on the book cover? A: The Amharic text on the book cover translates to English as follows:</p><p>\"Kill the righteous religion. Abune Tekle was a religious man and a farmer. When he saw them, they cut down their pine trees, put them in the blood and hid. When Balege was traveling from the countryside to the city, he was approached by Abune Takle Haimanot from the road.</p><p>Because he knows the nature of the man, he is the master of what he teaches. How are you, my brother, where is the prayer of Hajj? I am going to the city forever. Abune Tekleahimano and also a Hajj to Zyaw. When it's time to sleep, bro. We sat at the edge of this place, tasted grain and drank water.\"</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: Where is the Amharic text located on the book cover, and what does it translate to in English? A: The Amharic text is located in the middle section of the book cover. It translates to English as follows: \"Kill the righteous religion. Abune Tekle was a religious man and a farmer. When he saw them, they cut down their pine trees, put them in the blood and hid. When Balege was traveling from the countryside to the city, he was approached by Abune Takle Haimanot from the road. Because he knows the nature of the man, he is the master of what he teaches. How are you, my brother, where is the prayer of Hajj? I am going to the city forever. Abune Tekleahimano and also a Hajj to Zyaw. When it's time to sleep, bro. We sat at the edge of this place, tasted grain and drank water.\""</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>[repetitive text]</head><p>FFD score = 0.064267 D LLaVAR-2-VQA Examples Figure 11 and Figure 12 present examples in LLaVAR-2-VQA.</p><p>Extractive Q: Can you tell me more about John Belushi and why this book might be interesting? A: John Belushi was a renowned American comedian, actor, and musician, best known for his work on \"Saturday Night Live\" and his roles in films like \"Animal House\" and \"The Blues Brothers.\" He was a larger-than-life figure whose career was marked by both comedic brilliance and personal struggles, especially with substance abuse. \n\"Wired: The Short Life &amp; Fast Times of John Belushi\" by Bob Woodward provides an in-depth look at Belushi's tumultuous life, exploring his meteoric rise to fame and the challenges he faced, which ultimately led to his untimely death at the age of 33. The book is interesting because &#8230; &#8230;</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: What elements on the cover visually represent John Belushi? A: The image of John Belushi on the right side of the cover shows him holding a hat, wearing a blue shirt and jeans, with a yellow headband, which visually represents him.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>[irrelevant Self-explain]</head><p>FFD score = 1.070361</p><p>Extractive Q: Describe the design elements present on the book cover. A: The book cover is predominantly red with a diamond-shaped design in the center where the title \"Cold Hard Cache\" is written in white. The author's name, Amy Isaman, is displayed in smaller white font below the title. The cover features illustrations of keys, coins, and gold bars scattered around the edges, and a small illustration of a tower at the bottom center. There's also a banner at the top that reads \"A Tricia Seaver Mystery\".</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Self-explain</head><p>Q: How can you describe the design elements present on the book cover?" A: The book cover is predominantly red with a diamond-shaped design in the center where the title \"Cold Hard Cache\" is written in white. The author's name, Amy Isaman, is displayed in smaller white font below the title. The cover features illustrations of keys, coins, and gold bars scattered around the edges, and a small illustration of a tower at the bottom center. There's also a banner at the top that reads \"A Tricia Seaver Mystery\".</p><p>[</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>repetitive text]</head><p>FFD score = 0.152591 E Filtered out Examples Extractive data filtered out by mIFD score Figure 15 presents extractive examples filtered out using mIFD score. The analysis is included in each example. VQA examples filtered out by FFD score Figure 13 and Figure 14 present examples filtered out using FFD score including 2 cases: FFD score is closed to 0 and FFD score is closed to 1.</p></div></body>
		</text>
</TEI>
