More Like this

1. Multi-Modal Augmentation for Large Language Models with Applications to Task-Oriented Dialogues

   Samarinas, Chris; Promthaw, Pracha; Lekhwani, Rahul; Mysore, Sheshera; Huang, Sung Ming; Nijasure, Atharva; Zeng, Hansi; Zamani, Hamed (October 2023, 2nd Proceedings of Alexa Prize TaskBot (Alexa Prize 2023))

   We introduce MarunaBot V2, an advanced Task-Oriented Dialogue System (TODS) primarily aimed at aiding users in cooking and Do-It-Yourself tasks. We utilized large language models (LLMs) for data generation and inference, and implemented hybrid methods for intent classification, retrieval, and question answering, striking a balance between efficiency and performance. A key feature of our system is its multi-modal capabilities. We have incorporated a multi-modal enrichment technique that uses a fine-tuned CLIP model to supplement recipe instructions with pertinent images, a custom Diffusion model for image enhancement and generation, and a method for multi-modal option matching. A unique aspect of our system is its user-centric development approach, facilitated by a custom tool for tracking user interactions and swiftly integrating feedback. For a demonstration of our system, visit https://youtu.be/4MNI-puv_eE. 

   more » « less
2. Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation

   https://doi.org/10.1609/aaai.v39i12.33426  

   Ye, Yuyang; Zheng, Zhi; Shen, Yishan; Wang, Tianshu; Zhang, Hengruo; Zhu, Peijun; Yu, Runlong; Zhang, Kai; Xiong, Hui (April 2025, Proceedings of the AAAI Conference on Artificial Intelligence)

   Recent advances in Large Language Models (LLMs) have demonstrated significant potential in the field of Recommendation Systems (RSs). Most existing studies have focused on converting user behavior logs into textual prompts and leveraging techniques such as prompt tuning to enable LLMs for recommendation tasks. Meanwhile, research interest has recently grown in multimodal recommendation systems that integrate data from images, text, and other sources using modality fusion techniques. This introduces new challenges to the existing LLM-based recommendation paradigm which relies solely on text modality information. Moreover, although Multimodal Large Language Models (MLLMs) capable of processing multi-modal inputs have emerged, how to equip MLLMs with multi-modal recommendation capabilities remains largely unexplored. To this end, in this paper, we propose the Multimodal Large Language Model-enhanced Sequential Multimodal Recommendation (MLLM-MSR) model. To capture the dynamic user preference, we design a two-stage user preference summarization method. Specifically, we first utilize an MLLM-based item-summarizer to extract image feature given an item and convert the image into text. Then, we employ a recurrent user preference summarization generation paradigm to capture the dynamic changes in user preferences based on an LLM-based user-summarizer. Finally, to enable the MLLM for multi-modal recommendation task, we propose to fine-tune a MLLM-based recommender using Supervised Fine-Tuning (SFT) techniques. Extensive evaluations across various datasets validate the effectiveness of MLLM-MSR, showcasing its superior ability to capture and adapt to the evolving dynamics of user preferences. 

   more » « less
3. Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

   Li, Manling; Zhang, Lingyu; Radke, Richard J.; Ji, Heng (July 2019, 57th Conference of the Association for Computational Linguistics)

   Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models generate unfocused summaries. We develop an abstractive meeting summarizer from both videos and audios of meeting recordings. Specifically, we propose a multi-modal hierarchical attention mechanism across three levels: topic segment, utterance and word. To narrow down the focus into topically-relevant segments, we jointly model topic segmentation and summarization. In addition to traditional textual features, we introduce new multi-modal features derived from visual focus of attention, based on the assumption that an utterance is more important if its speaker receives more attention. Experiments show that our model significantly outperforms the state-of-the-art with both BLEU and ROUGE measures. 

   more » « less
4. Relation Learning on Social Networks with Multi-Modal Graph Edge Variational Autoencoders

   https://doi.org/10.1145/3336191.3371829  

   Yang, Carl; Zhang, Jieyu; Wang, Haonan; Li, Sha; Kim, Myungwan; Walker, Matt; Xiao, Yiou; Han, Jiawei (January 2020, WSDM'20: The Thirteenth ACM International Conference on Web Search and Data Mining)

   While node semantics have been extensively explored in social networks, little research attention has been paid to profile edge semantics, i.e., social relations. Ideal edge semantics should not only show that two users are connected, but also why they know each other and what they share in common. However, relations in social networks are often hard to profile, due to noisy multi-modal signals and limited user-generated ground-truth labels. In this work, we aim to develop a unified and principled framework that can profile user relations as edge semantics in social networks by integrating multi-modal signals in the presence of noisy and incomplete data. Our framework is also flexible towards limited or missing supervision. Specifically, we assume a latent distribution of multiple relations underlying each user link, and learn them with multi-modal graph edge variational autoencoders. We encode the network data with a graph convolutional network, and decode arbitrary signals with multiple reconstruction networks. Extensive experiments and case studies on two public DBLP author networks and two internal LinkedIn member networks demonstrate the superior effectiveness and efficiency of our proposed model. 

   more » « less
5. CONSISTENCY MODELS AS A RICH AND EFFICIENT POLICY CLASS FOR REINFORCEMENT LEARNING

   Ding, Zihan; Jin, Chi (May 2024, International Conference on Learning Representations)

   Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy. 

   more » « less