An official website of the United States government
Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these mod- els still suffer from some fundamental limitations related to robustness and gen- eralization due to the alignment and correlation between visual and textual fea- tures. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM’s pre- training and fine-tuning phases. In addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to re- construct the correct order of visual inputs. Then, we introduce a new Image-to- Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks.
more »
« less
This content will become publicly available on May 13, 2027
Discovering Urban Functional Zones using Training-Free Large Multi-Modal Models
Discovering urban functional zones (UFZs) is critical for understanding city spatial structures and supporting effective urban planning. Existing approaches to UFZ discovery typically rely on one of three costly strategies: (1) training large vision models directly on satellite imagery, which demands substantial computational resources; (2) leveraging crowdsourced data such as Points of Interest (POIs) from platforms like OpenStreetMap, which may be incomplete, inconsistent, or unavailable in many regions; or (3) collecting custom labeled data, which requires significant time, expense, and expert effort. Recently, large multi-modal models (LMMs) emerged as a promising alternative, offering strong capabilities in interpreting visual content without requiring extensive data labeling. However, their performance remains limited when applied to the UFZ discovery task, often struggling to capture the complex spatial and functional details and interactions of urban regions. To address this challenge, we propose a new approach that enhances LMMs’ reasoning capability to recognize urban functional zones by keeping LMM encoders frozen while training only lightweight graph-based models, eliminating the need for LMM fine-tuning or additional pre-training. Specifically, our approach first partitions the target area into small regions by road network, where an LMM is used for each region to generate visual and textual embeddings independently using its image and text encoders. Then, two graphs are constructed in which nodes represent regions with features defined by their respective embeddings, and edges encode their spatial adjacency. Message-passing on the two graphs hence captures spatial correlation between the visual and textual modalities. After that, graph clustering will suggest the prototypes representing nearby zones with similar urban functions, where contrastive learning is further leveraged to encourage cross-modal consistency. Evaluation on four city districts, namely Philadelphia (PA, USA), Pudong (Shanghai, China), San Francisco (CA, USA), and Seattle (WA, USA), substantiates the effectiveness of our proposal and that its performance is on par with supervised competitors.
more »
« less
- PAR ID:
- 10693491
- Publisher / Repository:
- ACM
- Date Published:
- Journal Name:
- ACM Transactions on Intelligent Systems and Technology
- ISSN:
- 2157-6904
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Graphs are ubiquitous structures found in numerous real-world applications, such as drug discovery, recommender systems, and social network analysis. To model graph-structured data, graph neural networks (GNNs) have become a popular tool. However, existing GNN architectures encounter challenges in cross-graph learning where multiple graphs have different feature spaces. To address this, recent approaches introduce text-attributed graphs (TAGs), where each node is associated with a textual description, which can be projected into a unified feature space using textual encoders. While promising, this method relies heavily on the availability of text-attributed graph data, which is difficult to obtain in practice. To bridge this gap, we propose a novel method named Topology-Aware Node description Synthesis (TANS), leveraging large language models (LLMs) to convert existing graphs into text-attributed graphs. The key idea is to integrate topological information into LLMs to explain how graph topology influences node semantics. We evaluate our TANS on text-rich, text-limited, and text-free graphs, demonstrating its applicability. Notably, on text-free graphs, our method significantly outperforms existing approaches that manually design node features, showcasing the potential of LLMs for preprocessing graph-structured data in the absence of textual information. The code and data are available at https://github.com/Zehong-Wang/TANS.more » « less
-
Recent advances in instruction tuning have led to the development of State-of-the-Art Large Multimodal Models (LMMs). Given the novelty of these models the impact of visual adversarial attacks on LMMs has not been thoroughly examined. We conduct a comprehensive study of the robustness of various LMMs against different adversarial attacks evaluated across tasks including image classification image captioning and Visual Question Answer (VQA). We find that in general LMMs are not robust to visual adversarial inputs. However our findings suggest that context provided to the model via prompts--such as questions in a QA pair--helps to mitigate the effects of visual adversarial inputs. Notably the LMMs evaluated demonstrated remarkable resilience to such attacks on the ScienceQA task with only an 8.10% drop in performance compared to their visual counterparts which dropped 99.73%. We also propose a new approach to real-world image classification which we term query decomposition. By incorporating existence queries into our input prompt we observe diminished attack effectiveness and improvements in image classification accuracy. This research highlights a previously under explored facet of LMM robustness and sets the stage for future work aimed at strengthening the resilience of multimodal systems in adversarial environments.more » « less
-
Wheeler, Travis (Ed.)Abstract MotivationDrug–target interaction (DTI) prediction accelerates drug discovery by identifying interactions between chemical compounds and proteins. Existing methods often rely on drug-drug and protein-protein similarity graphs but process them independently, limiting their ability to model interdependencies between modalities. Moving beyond isolated embedding generation from protein and drug graphs, we propose DCGAT-DTI, a novel deep learning framework with a dynamic cross-graph attention (DCGAT) module that dynamically models intra- and cross-graph interactions. Initial embeddings are generated using pretrained language models. Similarity graphs constructed from these embeddings are passed to DCGAT, which uses a Graph Convolutional Network-based Cross-Neighborhood Selection network to dynamically select cross-modal neighbors. This allows drug and protein embeddings to incorporate information from both modalities through intra- and cross-graph attention mechanisms. ResultsExtensive evaluations on four benchmark datasets demonstrate that DCGAT-DTI outperforms state-of-the-art methods across warm and cold start splits for both balanced and unbalanced datasets. In the challenging unbalanced cold start scenarios, it achieves significant improvement in performance for both drugs and proteins over the baselines. Availability and implementationSource code is available at https://github.com/compbiolabucf/DCGAT-DTI.more » « less
-
Generating molecular structures with desired properties is a critical task with broad applications in drug discovery and materials design. We propose 3M-Diffusion, a novel multi-modal molecular graph generation method, to generate diverse, ideally novel molecular structures with desired properties. 3M-Diffusion encodes molecular graphs into a graph latent space which it then aligns with the text space learned by encoder based LLMs from textual descriptions. It then reconstructs the molecular structure and atomic attributes based on the given text descriptions using the molecule decoder. It then learns a probabilistic mapping from the text space to the latent molecular graph space using a diffusion model. The results of our extensive experiments on several datasets demonstrate that 3M-Diffusion can generate high-quality, novel and diverse molecular graphs that semantically match the textual description provided. The code is available on github.more » « less
