More Like this

1. Retrieval as Attention: End-to-end Learning of Retrieval and Reading within a Single Transformer

   https://doi.org/10.18653/v1/2022.emnlp-main.149  

   Jiang, Zhengbao; Gao, Luyu; Wang, Zhiruo; Araki, Jun; Ding, Haibo; Callan, Jamie; Neubig, Graham (January 2022, Association for Computational Linguistics)

   Systems for knowledge-intensive tasks such as open-domain question answering (QA) usually consist of two stages: efficient retrieval of relevant documents from a large corpus and detailed reading of the selected documents. This is usually done through two separate models, a retriever that encodes the query and finds nearest neighbors, and a reader based on Transformers. These two components are usually modeled separately, which necessitates a cumbersome implementation and is awkward to optimize in an end-to-end fashion. In this paper, we revisit this design and eschew the separate architecture and training in favor of a single Transformer that performs retrieval as attention (RAA), and end-to-end training solely based on supervision from the end QA task. We demonstrate for the first time that an end-to-end trained single Transformer can achieve both competitive retrieval and QA performance on in-domain datasets, matching or even slightly outperforming state-of-the-art dense retrievers and readers. Moreover, end-to-end adaptation of our model significantly boosts its performance on out-of-domain datasets in both supervised and unsupervised settings, making our model a simple and adaptable end-to-end solution for knowledge-intensive tasks. 

   more » « less
2. Using Text to Teach Image Retrieval

   Haoyu Dong, Ze Wang (June 2021, CVPR 2021 Workshop)

   null (Ed.)
3. Using text to teach image retrieval

   https://doi.org/10.1109/CVPRW53098.2021.00180  

   H. Dong, Z. Wang (July 2021, CVPR 2021 Workshop)

   Image retrieval relies heavily on the quality of the data modeling and the distance measurement in the feature space. Building on the concept of image manifold, we first propose to represent the feature space of images, learned via neural networks, as a graph. Neighborhoods in the feature space are now defined by the geodesic distance between images, represented as graph vertices or manifold samples. When limited images are available, this manifold is sparsely sampled, making the geodesic computation and the corresponding retrieval harder. To address this, we augment the manifold samples with geometrically aligned text, thereby using a plethora of sentences to teach us about images. In addition to extensive results on standard datasets illustrating the power of text to help in image retrieval, a new public dataset based on CLEVR is introduced to quantify the semantic similarity between visual data and text data. The experimental results show that the joint embedding manifold is a robust representation, allowing it to be a better basis to perform image retrieval given only an image and a textual instruction on the desired modifications over the image. 

   more » « less
4. Using text to teach image retrieval

   H. Dong, Z. Wang (June 2021, CVPR 4th Multimodal Learning and Applications (MULA) Workshop)

   null (Ed.)
5. Using text to teach image retrieval

   H. Dong, Z. Wang (June 2021, CVPR 4th Multimodal Learning and Applications (MULA) Workshop)

   null (Ed.)