More Like this

1. Cross-Modal Center Loss for 3D Cross-Modal Retrieval

   Jing, L; Vahdani, E; Tan, J; Tian, Y. (June 2021, IEEE Conference on Computer Vision and Pattern Recognition (CVPR))

   null (Ed.)

   Cross-modal retrieval aims to learn discriminative and modal-invariant features for data from different modalities. Unlike the existing methods which usually learn from the features extracted by offline networks, in this paper, we pro- pose an approach to jointly train the components of cross- modal retrieval framework with metadata, and enable the network to find optimal features. The proposed end-to-end framework is updated with three loss functions: 1) a novel cross-modal center loss to eliminate cross-modal discrepancy, 2) cross-entropy loss to maximize inter-class variations, and 3) mean-square-error loss to reduce modality variations. In particular, our proposed cross-modal center loss minimizes the distances of features from objects belonging to the same class across all modalities. Extensive experiments have been conducted on the retrieval tasks across multi-modalities including 2D image, 3D point cloud and mesh data. The proposed framework significantly outperforms the state-of-the-art methods for both cross-modal and in-domain retrieval for 3D objects on the ModelNet10 and ModelNet40 datasets. 

   more » « less
2. From Multi-Modal Genotype and Phenotype Mutual Learning to Single-Modal Input for Longitudinal Outcome Prediction

   Ganjdanesh, Alireza; Zhang, Jipeng; Chen, Wei; Huang, Heng (January 2022, The 26th International Conference on Research in Computational Molecular Biology (RECOMB 2022))
3. Cross-modal 3D Shape Generation and Manipulation

   Cheng, Zezhou; Chai, Menglei; Ren, Jian; Lee, Hsin-Ying; Olszewski, Kyle; Huang, Zeng; Maji, Subhransu; Tulyakov, Sergey (November 2022, European Conference on Computer Vision (ECCV))
4. SUMMIT: Source-Free Adaptation of Uni-Modal Models to Multi-Modal Targets

   Cody Simons, Dripta S. (October 2023, IEEE/CVF Int. Conf. on Computer Vision)
5. Multi-Modal Geometric Learning for Grasping and Manipulation

   https://doi.org/10.1109/ICRA.2019.8794233  

   Watkins-Valls, David; Varley, Jacob; Allen, Peter (May 2019, 2019 International Conference on Robotics and Automation (ICRA))

   This work provides an architecture that incorporates depth and tactile information to create rich and accurate 3D models useful for robotic manipulation tasks. This is accomplished through the use of a 3D convolutional neural network (CNN). Offline, the network is provided with both depth and tactile information and trained to predict the object’s geometry, thus filling in regions of occlusion. At runtime, the network is provided a partial view of an object. Tactile information is acquired to augment the captured depth information. The network can then reason about the object’s geometry by utilizing both the collected tactile and depth information. We demonstrate that even small amounts of additional tactile information can be incredibly helpful in reasoning about object geometry. This is particularly true when information from depth alone fails to produce an accurate geometric prediction. Our method is benchmarked against and outperforms other visual-tactile approaches to general geometric reasoning. We also provide experimental results comparing grasping success with our method. 

   more » « less