More Like this

1. PhysioGMC: Generalizable Multi-modal Coordination for Physiological Signals

   Zhang, Mudi; Nakra, Anirudh; Wu, Min (August 2026, Proceedings of the 35th International Joint Conference on Artificial Intelligence (IJCAI-ECAI 2026) — Special Track on AI and Health)

   Physiological signals are widely used for health assessment in clinical and daily-life settings. Established physiological signals collected for in-patient and clinical use are often impractical for patients’ daily home use due to their complexity and resource demands. In contrast, wearable signals enable continuous monitoring in everyday life, but many have limited reliability and are not widely understood or accepted in clinical practices. To leverage complementary strengths of clinical and wearable physiological signals, we propose PhysioGMC, a Generalizable Multi-modal Coordination framework for Physiological signals that explicitly accounts for their strong inter-subject variability. PhysioGMC incorporates both clinical and wearable modalities into the training process to improve cross-subject performance when only a single wearable modality is available at deployment. The framework introduces a cross-modal contrastive learning module comprising two contrastive losses to jointly learn label-relevant, subject-agnostic representations across modalities. The self-supervised contrastive loss aligns latent features across modalities, while the supervised contrastive loss encourages learning label-discriminative features that are invariant to subject identity. Experiments on cardiovascular health monitoring and sleep staging tasks demonstrate that PhysioGMC consistently outperforms existing methods, achieving superior cross-subject performance at test time using only wearable modalities, such as photoplethysmography (PPG). 

   more » « less
2. A Deep Contrastive Learning Enabled Cross-Modal Retrieval for Cyber Manufacturing Services

   https://doi.org/10.1115/DETC2025-168949  

   Cook, Reagan; Yan, Xiaoliang; Melkote, Shreyes (August 2025, American Society of Mechanical Engineers)

   Abstract Cyber manufacturing is currently hindered by the lack of an efficient, automated system of retrieving part manufacturing information for instant quoting and process planning automation. Research efforts to date have demonstrated the efficacy of convolutional neural networks in capturing shape information, and of contrastive learning in classification-based retrieval and generation. However, cross-modal retrieval, which in the context of manufacturing enables retrieval of different part representations such as 2D engineering drawings and 3D CAD models commonly used in process and production planning, has not been widely studied. Addressing this gap could further efforts in cyber manufacturing services, leveraging a mixture of 2D or 3D part representations of previously produced parts similar in shape to the new query for process planning automation. To this end, this paper proposes training two deep-learning based encoder-decoder networks simultaneously and enforcing the similarity of their latent spaces to enable cross-modal retrieval. We present example retrieval results based on a synthetic dataset. The precision based on functional label demonstrates the model’s strength for use in manufacturer selection; the best model achieved 86.5% precision for retrieval of 3D objects given a 2D query, and 88.0% precision for retrieval of 2D objects given a 3D query. The model outperforms a classifier model used as a baseline. These results lay the groundwork for further research into cross-modal retrieval for cyber manufacturing. 

   more » « less
3. Self-supervised Feature Learning by Cross-modality and Cross-view Correspondences

   Jing, L; Zhang, L; Tian, Y (June 2021, The 4th Multimodal Learning and Applications (MULA) Workshop in conjunction with CVPR 2021)

   null (Ed.)

   The success of supervised learning requires large-scale ground truth labels which are very expensive, time- consuming, or may need special skills to annotate. To address this issue, many self- or un-supervised methods are developed. Unlike most existing self-supervised methods to learn only 2D image features or only 3D point cloud features, this paper presents a novel and effective self-supervised learning approach to jointly learn both 2D image features and 3D point cloud features by exploiting cross-modality and cross-view correspondences without using any human annotated labels. Specifically, 2D image features of rendered images from different views are extracted by a 2D convolutional neural network, and 3D point cloud features are extracted by a graph convolution neural network. Two types of features are fed into a two-layer fully connected neural network to estimate the cross-modality correspondence. The three networks are jointly trained (i.e. cross-modality) by verifying whether two sampled data of different modalities belong to the same object, meanwhile, the 2D convolutional neural network is additionally optimized through minimizing intra-object distance while maximizing inter-object distance of rendered images in different views (i.e. cross-view). The effectiveness of the learned 2D and 3D features is evaluated by transferring them on five different tasks including multi-view 2D shape recognition, 3D shape recognition, multi-view 2D shape retrieval, 3D shape retrieval, and 3D part-segmentation. Extensive evaluations on all the five different tasks across different datasets demonstrate strong generalization and effectiveness of the learned 2D and 3D features by the proposed self-supervised method. 

   more » « less
4. Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

   Zhiqiu Lin, Samuel Yu (June 2023, CVPR)

   The ability to quickly learn a new task with minimal instruction - known as few-shot learning - is a central aspect of intelligent agents. Classical few-shot benchmarks make use of few-shot samples from a single modality, but such samples may not be sufficient to characterize an entire concept class. In contrast, humans use cross-modal information to learn new concepts efficiently. In this work, we demonstrate that one can indeed build a better visual dog classifier by reading about dogs and listening to them bark. To do so, we exploit the fact that recent multimodal foundation models such as CLIP are inherently cross-modal, mapping different modalities to the same representation space. Specifically, we propose a simple cross-modal adaptation approach that learns from few-shot examples spanning different modalities. By repurposing class names as additional one-shot training samples, we achieve SOTA results with an embarrassingly simple linear classifier for vision-language adaptation. Furthermore, we show that our approach can benefit existing methods such as prefix tuning, adapters, and classifier ensembling. Finally, to explore other modalities beyond vision and language, we construct the first (to our knowledge) audiovisual few-shot benchmark and use cross-modal training to improve the performance of both image and audio classification. 

   more » « less
5. USING DEEP CROSS MODAL HASHING AND ERROR CORRECTING CODES FOR IMPROVING THE EFFICIENCY OF ATTRIBUTE GUIDED FACIAL IMAGE RETRIEVAL

   https://doi.org/10.1109/GlobalSIP.2018.8646467  

   Talreja, Veeru; Taherkhani, Fariborz; Valenti, Matthew C.; Nasrabadi, Nasser M. (November 2018, IEEE Global Conference on Signal and Information Processing (GlobalSIP))

   With benefits of fast query speed and low storage cost, hashing-based image retrieval approaches have garnered considerable attention from the research community. In this paper, we propose a novel Error-Corrected Deep Cross Modal Hashing (CMH-ECC) method which uses a bitmap specifying the presence of certain facial attributes as an input query to retrieve relevant face images from the database. In this architecture, we generate compact hash codes using an end-to-end deep learning module, which effectively captures the inherent relationships between the face and attribute modality. We also integrate our deep learning module with forward error correction codes to further reduce the distance between different modalities of the same subject. Specifically, the properties of deep hashing and forward error correction codes are exploited to design a cross modal hashing framework with high retrieval performance. Experimental results using two standard datasets with facial attributes-image modalities indicate that our CMH-ECC face image retrieval model outperforms most of the current attribute-based face image retrieval approaches. 

   more » « less