More Like this

1. A Joint Speaker-Listener-Reinforcer Model for Referring Expressions

   Licheng Yu, Hao Tan (January 2017, IEEE Conference on Computer Vision and Pattern Recognition)

   Referring expressions are natural language construc- tions used to identify particular objects within a scene. In this paper, we propose a unified framework for the tasks of referring expression comprehension and generation. Our model is composed of three modules: speaker, listener, and reinforcer. The speaker generates referring expressions, the listener comprehends referring expressions, and the rein- forcer introduces a reward function to guide sampling of more discriminative expressions. The listener-speaker mod- ules are trained jointly in an end-to-end learning frame- work, allowing the modules to be aware of one another during learning while also benefiting from the discrimina- tive reinforcer’s feedback. We demonstrate that this unified framework and training achieves state-of-the-art results for both comprehension and generation on three referring ex- pression datasets. 

   more » « less
2. Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

   Dou, Zi-Yi; Kamath, Aishwarya; Gan, Zhe; Zhang, Pengchuan; Wang, Jianfeng; Li, Linjie; Liu, Zicheng; Liu, Ce; LeCun, Yann; Peng, Nanyun; et al (October 2022, NeurIPS)

   Vision-language (VL) pre-training has recently received considerable attention. However, most existing end-to-end pre-training approaches either only aim to tackle VL tasks such as image-text retrieval, visual question answering (VQA) and image captioning that test high-level understanding of images, or only target region-level understanding for tasks such as phrase grounding and object detection. We present FIBER (Fusion-In-the-Backbone-based transformER), a new VL model architecture that can seamlessly handle both these types of tasks. Instead of having dedicated transformer layers for fusion after the uni-modal backbones, FIBER pushes multimodal fusion deep into the model by inserting cross-attention into the image and text backbones to better capture multimodal interactions. In addition, unlike previous work that is either only pre-trained on image-text data or on fine-grained data with box-level annotations, we present a two-stage pre-training strategy that uses both these kinds of data efficiently: (i) coarse-grained pre-training based on image-text data; followed by (ii) fine-grained pre-training based on image-text-box data. We conduct comprehensive experiments on a wide range of VL tasks, ranging from VQA, image captioning, and retrieval, to phrase grounding, referring expression comprehension, and object detection. Using deep multimodal fusion coupled with the two-stage pre-training, FIBER provides consistent performance improvements over strong baselines across all tasks, often outperforming methods using magnitudes more data. Code is released at https://github.com/microsoft/FIBER. 

   more » « less
3. Engineering DNA recognition and allosteric response properties of TetR family proteins by using a module-swapping strategy

   https://doi.org/10.1093/nar/gkz666  

   Dimas, Rey P; Jordan, Benjamin R; Jiang, Xian-Li; Martini, Catherine; Glavy, Joseph S; Patterson, Dustin P; Morcos, Faruck; Chan, Clement T (August 2019, Nucleic Acids Research)

   Abstract The development of synthetic biological systems requires modular biomolecular components to flexibly alter response pathways. In previous studies, we have established a module-swapping design principle to engineer allosteric response and DNA recognition properties among regulators in the LacI family, in which the engineered regulators served as effective components for implementing new cellular behavior. Here we introduced this protein engineering strategy to two regulators in the TetR family: TetR (UniProt Accession ID: P04483) and MphR (Q9EVJ6). The TetR DNA-binding module and the MphR ligand-binding module were used to create the TetR-MphR. This resulting hybrid regulator possesses DNA-binding properties of TetR and ligand response properties of MphR, which is able to control gene expression in response to a molecular signal in cells. Furthermore, we studied molecular interactions between the TetR DNA-binding module and MphR ligand-binding module by using mutant analysis. Together, we demonstrated that TetR family regulators contain discrete and functional modules that can be used to build biological components with novel properties. This work highlights the utility of rational design as a means of creating modular parts for cell engineering and introduces new possibilities in rewiring cellular response pathways. 

   more » « less
4. HUMAN-CENTERED EMOTION RECOGNITION IN ANIMATED GIFS

   Yang, Zhengyuan; Zhang, Yixuan; Luo, Jiebo (January 2019, Proceedings of IEEE International Conference on Multimedia and Expo)

   As an intuitive way of expression emotion, the animated Graphical Interchange Format (GIF) images have been widely used on social media. Most previous studies on automated GIF emotion recognition fail to effectively utilize GIF’s unique properties, and this potentially limits the recognition performance. In this study, we demonstrate the importance of human related information in GIFs and conduct humancentered GIF emotion recognition with a proposed Keypoint Attended Visual Attention Network (KAVAN). The framework consists of a facial attention module and a hierarchical segment temporal module. The facial attention module exploits the strong relationship between GIF contents and human characters, and extracts frame-level visual feature with a focus on human faces. The Hierarchical Segment LSTM (HSLSTM) module is then proposed to better learn global GIF representations. Our proposed framework outperforms the state-of-the-art on the MIT GIFGIF dataset. Furthermore, the facial attention module provides reliable facial region mask predictions, which improves the model’s interpretability. 

   more » « less
5. Using Co-Captured Face, Gaze, and Verbal Reactions to Images of Varying Emotional Content for Analysis and Semantic Alignment

   Gangji, Aliya; Walden, Trevor; Vaidyanathan, Preethi; Prud'hommeaux, Emily; Bailey, Reynold; Alm, Cecilia O (January 2017, The AAAI-17 Workshop on Human-Aware Artificial Intelligence)

   Analyzing different modalities of expression can provide insights into the ways that humans interpret, label, and react to images. Such insights have the potential not only to advance our understanding of how humans coordinate these expressive modalities but also to enhance existing methodologies for common AI tasks such as image annotation and classification. We conducted an experiment that co-captured the facial expressions, eye movements, and spoken language data that observers produce while examining images of varying emotional content and responding to description-oriented vs. affect-oriented questions about those images. We analyzed the facial expressions produced by the observers in order to determine the connection between those expressions and an image's emotional content. We also explored the relationship between the valence of an image and the verbal responses to that image, and how that relationship relates to the nature of the prompt, using low-level lexical features and more complex affective features extracted from the observers' verbal responses. Finally, in order to integrate this multimodal data, we extended an existing bitext alignment framework to create meaningful pairings between narrated observations about images and the image regions indicated by eye movement data. The resulting annotations of image regions with words from observers' responses demonstrate the potential of bitext alignment for multimodal data integration and, from an application perspective, for annotation of open-domain images. In addition, we found that while responses to affect-oriented questions appear useful for image understanding, their holistic nature seems less helpful for image region annotation. 

   more » « less