More Like this

1. Towards Addressing the Misalignment of Object Proposal Evaluation for Vision-Language Tasks via Semantic Grounding

   Feinglass, Joshua; Yang, Yezhou (January 2024, Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024)

   Object proposal generation serves as a standard pre-processing step in Vision-Language (VL) tasks (image captioning, visual question answering, etc.). The performance of object proposals generated for VL tasks is currently evaluated across all available annotations, a protocol that we show is misaligned - higher scores do not necessarily correspond to improved performance on downstream VL tasks. Our work serves as a study of this phenomenon and explores the effectiveness of semantic grounding to mitigate its effects. To this end, we propose evaluating object proposals against only a subset of available annotations, selected by thresholding an annotation importance score. Importance of object annotations to VL tasks is quantified by extracting relevant semantic information from text describing the image. We show that our method is consistent and demonstrates greatly improved alignment with annotations selected by image captioning metrics and human annotation when compared against existing techniques. Lastly, we compare current detectors used in the Scene Graph Generation (SGG) benchmark as a use case, which serves as an example of when traditional object proposal evaluation techniques are misaligned. 

   more » « less
2. No Head Left Behind - Multi-Head Alignment Distillation for Transformers

   Zhao, T; Singh, K Y; Appalaraju, S; Tang, P; Mahadevan, V; Manmatha, R; Wu, Y N (February 2024, AAAI Conference on Artificial Intelligence)

   Knowledge distillation aims at reducing model size without compromising much performance. Recent work has applied it to large vision-language (VL) Transformers, and has shown that attention maps in the multi-head attention modules of vision-language Transformers contain extensive intra-modal and cross-modal co-reference relations to be distilled. The standard approach is to apply a one-to-one attention map distillation loss, i.e. the Teacher’s first attention head instructs the Student’s first head, the second teaches the second, and so forth, but this only works when the numbers of attention heads in the Teacher and Student are the same. To remove this constraint, we propose a new Attention Map Alignment Distillation (AMAD) method for Transformers with multi-head attention, which works for a Teacher and a Student with different numbers of attention heads. Specifically, we soft-align different heads in Teacher and Student attention maps using a cosine similarity weighting. The Teacher head contributes more to the Student heads for which it has a higher similarity weight. Each Teacher head contributes to all the Student heads by minimizing the divergence between the attention activation distributions for the soft-aligned heads. No head is left behind. This distillation approach operates like cross-attention. We experiment on distilling VL-T5 and BLIP, and apply AMAD loss on their T5, BERT, and ViT sub-modules. We show, under vision-language setting, that AMAD outperforms conventional distillation methods on VQA-2.0, COCO captioning, and Multi30K translation datasets. We further show that even without VL pre-training, the distilled VL-T5 models outperform corresponding VL pre-trained VL-T5 models that are further fine-tuned by ground-truth signals, and that fine-tuning distillation can also compensate to some degree for the absence of VL pre-training for BLIP models. 

   more » « less
3. VQAttack: Transferable Adversarial Attacks on Visual Question Answering via Pre-trained Models

   https://doi.org/10.1609/aaai.v38i7.28499  

   Yin, Ziyi; Ye, Muchao; Zhang, Tianrong; Wang, Jiaqi; Liu, Han; Chen, Jinghui; Wang, Ting; Ma, Fenglong (March 2024, Proceedings of the AAAI Conference on Artificial Intelligence)

   Visual Question Answering (VQA) is a fundamental task in computer vision and natural language process fields. Although the “pre-training & finetuning” learning paradigm significantly improves the VQA performance, the adversarial robustness of such a learning paradigm has not been explored. In this paper, we delve into a new problem: using a pre-trained multimodal source model to create adversarial image-text pairs and then transferring them to attack the target VQA models. Correspondingly, we propose a novel VQATTACK model, which can iteratively generate both im- age and text perturbations with the designed modules: the large language model (LLM)-enhanced image attack and the cross-modal joint attack module. At each iteration, the LLM-enhanced image attack module first optimizes the latent representation-based loss to generate feature-level image perturbations. Then it incorporates an LLM to further enhance the image perturbations by optimizing the designed masked answer anti-recovery loss. The cross-modal joint attack module will be triggered at a specific iteration, which updates the image and text perturbations sequentially. Notably, the text perturbation updates are based on both the learned gradients in the word embedding space and word synonym-based substitution. Experimental results on two VQA datasets with five validated models demonstrate the effectiveness of the proposed VQATTACK in the transferable attack setting, compared with state-of-the-art baselines. This work revealsa significant blind spot in the “pre-training & fine-tuning” paradigm on VQA tasks. The source code can be found in the link https://github.com/ericyinyzy/VQAttack. 

   more » « less
4. From Literature to Classifier: Automated Defect Classification in Manufacturing via LLM-based Knowledge Transfer

   https://doi.org/10.2139/ssrn.6458658  

   Naghavi_Khanghah, Kiarash; Patel, Anandkumar; Yang, Qian; Malhotra, Rajiv; Xu, Hongyi (March 2026, SSRN)

   Defect detection and classification in manufacturing systems is an important task in ensuring the quality of manufactured parts. However, conventional deep learning approaches require extensive datasets that are expensive and time-consuming to acquire, particularly when labeled data is necessary. Similarly, large language model (LLM)-based techniques also require high-quality labeled data for few-shot or fine-tuning, whereas zero-shot methods lack domain-specific knowledge for defect classification. This work presents a multimodal framework that can autonomously derive domain knowledge from scientific literature and transform it into multimodal training sets and knowledge representations without the need for pre-labeled data. Three workflows, each consisting of their own knowledge extraction modules, are employed: (1) a text-based approach using Knowledge Augmented Generation (KAG) for structured reasoning about defect characteristics, (2) an image-based model using multi-stage retrieval to extract single-defect reference images from scientific Literature, and (3) a Transformer-based Variational Autoencoder, which does not require any external training data. Each classification method operates independently before the ensemble decision fusion combines all three approaches, enabling both understanding of the capabilities of the individual method and the relative merits of the ensemble in combining the strengths. Although the proposed framework is process-agnostic, we demonstrate it on Fused Deposition Modeling (FDM) as a representative testbed with four common defect categories (over extrusion, under extrusion, warping, and spaghetti). The ensemble fusion approach obtains a mean accuracy of 78.9%, which is on average 8.9% relative improvement over individual methods (11.4% relative improvement over image-based methods), validating the effectiveness of multi-modal fusion in text-based reasoning, image comparison, and representation learning. This research proposes a framework for manufacturing quality control by transforming scientific knowledge into high-quality training datasets and then using it within advanced classification systems, showing potential applicability to data-scarce manufacturing domains. 

   more » « less
5. CoRank: LLM-Based Compact Reranking with Document Features for Scientific Retrieval

   Tian, Runchu; Xu, Xueqiang; Jin, Bowen; Kang, SeongKu; Han, Jiawei (August 2026, ACM: Proc. 2026 ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD 2026))

   Scientific retrieval is essential for advancing scientific knowledge discovery. Within this process, document reranking plays a critical role in refining first-stage retrieval results. Recently, LLMs have become popular reranking backbones for their strong domain knowledge. However, standard LLM listwise reranking faces challenges in the scientific domain. First-stage retrieval is often suboptimal in the scientific domain, so relevant documents are ranked lower. Meanwhile, conventional listwise reranking places the full text of candidates into the context window, limiting the number of candidates that can be considered. As a result, many relevant documents are excluded before reranking, constraining overall retrieval performance. To address these challenges, we explore semantic-feature-based compact document representations (e.g., categories, sections, and keywords) and propose CoRank, a training-free, model-agnostic reranking framework for scientific retrieval. It presents a three-stage solution: (i) offline extraction of document features, (ii) coarse-grained reranking using these compact representations, and (iii) fine-grained reranking on full texts of the top candidates from (ii). This integrated process addresses suboptimal first-stage retrieval: Compact representations allow more documents to fit within the context window, improving candidate set coverage, while the final fine-grained ranking ensures a more accurate ordering. Experiments on 5 academic retrieval datasets show that CoRank significantly improves reranking performance across different LLM backbones (average nDCG@10 from 50.6 to 55.5). 

   more » « less