An official website of the United States government
As more and more search traffic comes from mobile phones, intelligent assistants, and smart-home devices, new challenges (e.g., limited presentation space) and opportunities come up in information retrieval. Previously, an effective technique, relevance feedback (RF), has rarely been used in real search scenarios due to the overhead of collecting users’ relevance judgments. However, since users tend to interact more with the search results shown on the new interfaces, it becomes feasible to obtain users’ assessments on a few results during each interaction. This makes iterative relevance feedback (IRF) techniques look promising today. IRF can deal with a simplified scenario of conversational search, where the system asks users to provide relevance feedback on results shown in the current iteration and shows more relevant results in the next interaction. IRF has not been studied systematically in the new search scenarios and its effectiveness is mostly unknown. In this paper, we re-visit IRF and extend it with RF models proposed in recent years. We conduct extensive experiments to analyze and compare IRF with the standard top-k RF framework on document and passage retrieval. Experimental results show that IRF is at least as effective as the standard top-k RF framework for documents and much more effective for passages. This indicates that IRF for passage retrieval has huge potential and is a promising direction for conversational search based on relevance feedback.
more »
« less
A Reinforcement Learning Framework for Relevance Feedback
Relevance feedback is an effective technique for improving retrieval performance using the feedback documents. Selecting effective feedback terms and weighting them have been always challenging. Several methods based on different assumptions have been so far proposed, however, they do not directly optimize the retrieval performance. Learning an effective relevance feedback model is not trivial since the true feedback distribution is unknown. In this paper, we propose a general reinforcement learning framework for relevance feedback, called RML. Our framework directly optimizes any desired retrieval metric, including precision-oriented, recall-oriented, and even diversity metrics. RML can be easily extended to directly optimize any arbitrary user satisfaction signal. Experiments on standard TREC collections demonstrate the effectiveness of our framework.
more »
« less
- Award ID(s):
- 1617408
- PAR ID:
- 10175982
- Date Published:
- Journal Name:
- Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR2020)
- Page Range / eLocation ID:
- 59 to 68
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Retrieval-augmented generation (RAG) systems can effectively address user queries by leveraging indexed document corpora to retrieve the relevant contexts. Ranking techniques have been adopted in RAG systems to sort the retrieved contexts by their relevance to the query so that users can select the most useful contexts for their downstream tasks. While many existing ranking methods rely on the similarity between the embedding vectors of the context and query to measure relevance, it is important to note that similarity does not equate to relevance in all scenarios. Some ranking methods use large language models (LLMs) to rank the contexts by putting the query and the candidate contexts in the prompt and asking LLM about their relevance. The scalability of those methods is contingent on the number of candidate contexts and the context window of those LLMs. Also, those methods require fine-tuning the LLMs, which can be computationally expensive and require domain-related data. In this work, we propose a scalable ranking framework that does not involve LLM training. Our framework uses an off-the-shelf LLM to hypothesize the user's query based on the retrieved contexts and ranks the contexts based on the similarity between the hypothesized queries and the user query. Our framework is efficient at inference time and is compatible with many other context retrieval and ranking techniques. Experimental results show that our method improves the ranking performance of retrieval systems in multiple benchmarks.more » « less
-
A prevalent approach of entity-oriented systems involves retrieving relevant entities by harnessing knowledge graph embeddings. These embeddings encode entity information in the context of the knowledge graph and are static in nature. Our goal is to generate entity embeddings that capture what renders them relevant for the query. This differs from entity embeddings constructed with static resource, for example, E-BERT. Previously, ~\citet{dalton2014entity} demonstrated the benefits obtained with the Entity Context Model, a pseudo-relevance feedback approach based on entity links in relevant contexts. In this work, we reinvent the Entity Context Model (ECM) for neural graph networks and incorporate pre-trained embeddings. We introduce three entity ranking models based on fundamental principles of ECM: (1) \acl{GAN}, (2) Simple Graph Relevance Networks, and (3) Graph Relevance Networks. \acl{GAN} and Graph Relevance Networks are the graph neural variants of ECM, that employ attention mechanism and relevance information of the relevant context respectively to ascertain entity relevance. Our experiments demonstrate that our neural variants of the ECM model significantly outperform the state-of-the-art BERT-ER ~\cite{10.1145/3477495.3531944} by more than 14\% and exceeds the performance of systems that use knowledge graph embeddings by over 101\%. Notably, our findings reveal that leveraging the relevance of the relevant context is more effective at identifying relevant entities than the attention mechanism. To evaluate the efficacy of the models, we conduct experiments on two standard benchmark datasets, DBpediaV2 and TREC Complex Answer Retrieval. To aid reproducibility, our code and data are available. https://github.com/TREMA-UNH/neural-entity-context-modelsmore » « less
-
Satellite sensors have been widely used for precipitation retrieval, and a number of precipitation retrieval algorithms have been developed using observations from various satellite sensors. The current operational rainfall rate quantitative precipitation estimate (RRQPE) product from the geostationary operational environmental satellite (GOES) offers full disk rainfall rate estimates based on the observations from the advanced baseline imager (ABI) aboard the GOES-R series. However, accurate precipitation retrieval using satellite sensors is still challenging due to the limitations on spatio-temporal sampling of the satellite sensors and/or the uncertainty associated with the applied parametric retrieval algorithms. In this article, we propose a deep learning framework for precipitation retrieval using the combined observations from the ABI and geostationary lightning mapper (GLM) on the GOES-R series to improve the current operational RRQPE product. Particularly, the proposed deep learning framework is composed of two deep convolutional neural networks (CNNs) that are designed for precipitation detection and quantification. The cloud-top brightness temperature from multiple ABI channels and the lightning flash rate from the GLM measurement are used as inputs to the deep learning framework. To train the designed CNNs, the precipitation product multiradar multi-sensor (MRMS) system from the National Oceanic and Atmospheric Administration (NOAA) is used as target labels to optimize the network parameters. The experimental results show that the precipitation retrieval performance of the proposed framework is superior to the currently operational GOES RRQPE product in the selected study domain, and the performance is dramatically enhanced after incorporating the lightning data into the deep learning model. Using the independent MRMS product as a reference, the deep learning model can reduce the retrieval uncertainty in the operational RRQPE product by at least 31% in terms of the mean squared error and normalized mean absolute error, and the improvement is more significant in moderate to heavy rain regions. Therefore, the proposed deep learning framework can potentially serve as an alternative approach for GOES precipitation retrievals.more » « less
-
Relevance feedback techniques assume that users provide relevance judgments for the top k (usually 10) documents and then re-rank using a new query model based on those judgments. Even though this is effective, there has been little research recently on this topic because requiring users to provide substantial feedback on a result list is impractical in a typical web search scenario. In new environments such as voice-based search with smart home devices, however, feedback about result quality can potentially be obtained during users' interactions with the system. Since there are severe limitations on the length and number of results that can be presented in a single interaction in this environment, the focus should move from browsing result lists to iterative retrieval and from retrieving documents to retrieving answers. In this paper, we study iterative relevance feedback techniques with a focus on retrieving answer passages. We first show that iterative feedback can be at least as effective as the top-k approach on standard TREC collections, and more effective on answer passage collections. We then propose an iterative feedback model for answer passages based on semantic similarity at passage level and show that it can produce significant improvements compared to both word-based iterative feedback models and those based on term-level semantic similarity.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1145/3397271.3401099
