More Like this

1. Husformer: A Multimodal Transformer for Multimodal Human State Recognition

   https://doi.org/10.1109/TCDS.2024.3357618  

   Wang, Ruiqi; Jo, Wonse; Zhao, Dezhong; Wang, Weizheng; Gupte, Arjun; Yang, Baijian; Chen, Guohua; Min, Byung-Cheol (January 2024, IEEE Transactions on Cognitive and Developmental Systems)

   Human state recognition is a critical topic with pervasive and important applications in human–machine systems. Multimodal fusion, which entails integrating metrics from various data sources, has proven to be a potent method for boosting recognition performance. Although recent multimodal-based models have shown promising results, they often fall short in fully leveraging sophisticated fusion strategies essential for modeling adequate cross-modal dependencies in the fusion representation. Instead, they rely on costly and inconsistent feature crafting and alignment. To address this limitation, we propose an end-to-end multimodal transformer framework for multimodal human state recognition called Husformer. Specifically, we propose using cross-modal transformers, which inspire one modality to reinforce itself through directly attending to latent relevance revealed in other modalities, to fuse different modalities while ensuring sufficient awareness of the cross-modal interactions introduced. Subsequently, we utilize a self-attention transformer to further prioritize contextual information in the fusion representation. Extensive experiments on two human emotion corpora (DEAP and WESAD) and two cognitive load datasets [multimodal dataset for objective cognitive workload assessment on simultaneous tasks (MOCAS) and CogLoad] demonstrate that in the recognition of the human state, our Husformer outperforms both state-of-the-art multimodal baselines and the use of a single modality by a large margin, especially when dealing with raw multimodal features. We also conducted an ablation study to show the benefits of each component in Husformer. Experimental details and source code are available at https://github.com/SMARTlab-Purdue/Husformer. 

   more » « less
2. Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation

   https://doi.org/10.1609/aaai.v39i12.33426  

   Ye, Yuyang; Zheng, Zhi; Shen, Yishan; Wang, Tianshu; Zhang, Hengruo; Zhu, Peijun; Yu, Runlong; Zhang, Kai; Xiong, Hui (April 2025, Proceedings of the AAAI Conference on Artificial Intelligence)

   Recent advances in Large Language Models (LLMs) have demonstrated significant potential in the field of Recommendation Systems (RSs). Most existing studies have focused on converting user behavior logs into textual prompts and leveraging techniques such as prompt tuning to enable LLMs for recommendation tasks. Meanwhile, research interest has recently grown in multimodal recommendation systems that integrate data from images, text, and other sources using modality fusion techniques. This introduces new challenges to the existing LLM-based recommendation paradigm which relies solely on text modality information. Moreover, although Multimodal Large Language Models (MLLMs) capable of processing multi-modal inputs have emerged, how to equip MLLMs with multi-modal recommendation capabilities remains largely unexplored. To this end, in this paper, we propose the Multimodal Large Language Model-enhanced Sequential Multimodal Recommendation (MLLM-MSR) model. To capture the dynamic user preference, we design a two-stage user preference summarization method. Specifically, we first utilize an MLLM-based item-summarizer to extract image feature given an item and convert the image into text. Then, we employ a recurrent user preference summarization generation paradigm to capture the dynamic changes in user preferences based on an LLM-based user-summarizer. Finally, to enable the MLLM for multi-modal recommendation task, we propose to fine-tune a MLLM-based recommender using Supervised Fine-Tuning (SFT) techniques. Extensive evaluations across various datasets validate the effectiveness of MLLM-MSR, showcasing its superior ability to capture and adapt to the evolving dynamics of user preferences. 

   more » « less
3. Multimodal Belief Prediction

   https://doi.org/10.21437/Interspeech.2024-2103  

   Murzaku, John; Soubki, Adil; Rambow, Owen (September 2024, ISCA)
4. Multimodal Remote Inference

   https://doi.org/10.1109/MASS66014.2025.00039  

   Zhang, Keyuan; Sun, Yin; Ji, Bo (October 2025, IEEE)
5. Multimodal Routing: Improving Local and Global Interpretability of Multimodal Language Analysis

   Tsai, Y.-H.; Ma, M.; Yang, M.; Salakhutdinov, R.; Morency, L.-P. (January 2020, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP))