More Like this

1. Mixed Modality Workflows in TaskVine

   https://doi.org/10.1145/3588195.3595953  

   Simonetti, David; Tovar, Ben; Thain, Douglas (August 2023, ACM)

   Modern scientific workflows desire to mix several different comput- ing modalities: self-contained computational tasks, data-intensive transformations, and serverless function calls. To date, these modali- ties have required distinct system architectures with different sched- uling objectives and constraints. In this paper, we describe how TaskVine, a new workflow execution platform, combines these modalities into an execution platform with shared abstractions. We demonstrate results of the system executing a machine learning workflow with combined standalone tasks and serverless functions. 

   more » « less
2. Differential processing in modality-specific Mauthner cell dendrites: Modality-dependent dendrite specialization in the M-cell

   https://doi.org/10.1113/JP274861  

   Medan, Violeta; Mäki-Marttunen, Tuomo; Sztarker, Julieta; Preuss, Thomas (February 2018, The Journal of Physiology)
3. Co-Modality Imbalanced Graph Contrastive Learning

   Qian, Yiyue; Zhang, Chuihui; Zhang, Yiming; Wen, Qianlong; Ye, Yanfang; Zhang, Chuxu (November 2022, Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS))
4. A Dependency Structure Annotation for Modality

   https://doi.org/10.18653/v1/W19-3321  

   Vigus, Meagan; Van Gysel, Jens E.; Croft, William (January 2019, Proceedings of the First International Workshop on Designing Meaning Representations (DMR 2019))

   Xue, Nianwen; Croft, William; Hajic, Jan; Huang, Chu-Ren; Oepen, Stephan; Palmer, Martha; Pustejovsky, James (Ed.)

   This paper presents an annotation scheme for modality that employs a dependency structure. Events and sources (here, conceivers) are represented as nodes and epistemic strength relations characterize the edges. The epistemic strength values are largely based on Saurí and Pustejovsky’s (2009) FactBank, while the dependency structure mirrors Zhang and Xue’s (2018b) approach to temporal relations. Six documents containing 377 events have been annotated by two expert annotators with high levels of agreement. 

   more » « less
5. Modality-Balanced Models for Visual Dialogue

   https://doi.org/10.1609/aaai.v34i05.6320  

   Kim, Hyounghun; Tan, Hao; Bansal, Mohit (February 2020, The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI-20))

   The Visual Dialog task requires a model to exploit both im- age and conversational context information to generate the next response to the dialogue. However, via manual analysis, we find that a large number of conversational questions can be answered by only looking at the image without any access to the context history, while others still need the conversa- tion context to predict the correct answers. We demonstrate that due to this reason, previous joint-modality (history and image) models over-rely on and are more prone to memoriz- ing the dialogue history (e.g., by extracting certain keywords or patterns in the context information), whereas image-only models are more generalizable (because they cannot memo- rize or extract keywords from history) and perform substan- tially better at the primary normalized discounted cumula- tive gain (NDCG) task metric which allows multiple correct answers. Hence, this observation encourages us to explic- itly maintain two models, i.e., an image-only model and an image-history joint model, and combine their complementary abilities for a more balanced multimodal model. We present multiple methods for this integration of the two models, via ensemble and consensus dropout fusion with shared param- eters. Empirically, our models achieve strong results on the Visual Dialog challenge 2019 (rank 3 on NDCG and high bal- ance across metrics), and substantially outperform the winner of the Visual Dialog challenge 2018 on most metrics. 

   more » « less