An official website of the United States government
Retrieval-augmented generation (RAG) systems can effectively address user queries by leveraging indexed document corpora to retrieve the relevant contexts. Ranking techniques have been adopted in RAG systems to sort the retrieved contexts by their relevance to the query so that users can select the most useful contexts for their downstream tasks. While many existing ranking methods rely on the similarity between the embedding vectors of the context and query to measure relevance, it is important to note that similarity does not equate to relevance in all scenarios. Some ranking methods use large language models (LLMs) to rank the contexts by putting the query and the candidate contexts in the prompt and asking LLM about their relevance. The scalability of those methods is contingent on the number of candidate contexts and the context window of those LLMs. Also, those methods require fine-tuning the LLMs, which can be computationally expensive and require domain-related data. In this work, we propose a scalable ranking framework that does not involve LLM training. Our framework uses an off-the-shelf LLM to hypothesize the user's query based on the retrieved contexts and ranks the contexts based on the similarity between the hypothesized queries and the user query. Our framework is efficient at inference time and is compatible with many other context retrieval and ranking techniques. Experimental results show that our method improves the ranking performance of retrieval systems in multiple benchmarks.
more »
« less
Problem solving is embedded in context… so how do we measure it?
Problem solving encompasses the broad domain of human, goal-directed behaviors. Though we may attempt to measure problem solving using tightly controlled and decontextualized tasks, it is inextricably embedded in both reasoners’ experiences and their contexts. Without situating problem solvers, problem contexts, and our own experiential partialities as researchers, we risk intertwining the research of information relevance with our own confirmatory biases about people, environments, and ourselves. We review each of these ecological facets of information relevance in problem solving, and we suggest a framework to guide its measurement. We ground this framework with concrete examples of ecologically valid, culturally relevant measurement of problem solving.
more »
« less
- Award ID(s):
- 2141411
- PAR ID:
- 10538848
- Publisher / Repository:
- Frontiers
- Date Published:
- Journal Name:
- Frontiers in Psychology
- Volume:
- 15
- ISSN:
- 1664-1078
- Subject(s) / Keyword(s):
- problem solving, measurement, information relevance, ecological validity, cultural relevance
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Abstract Many models of evolution are implicitly causal processes. Features such as causal feedback between evolutionary variables and evolutionary processes acting at multiple levels, though, mean that conventional causal models miss important phenomena. We develop here a general theoretical framework for analyzing evolutionary processes drawing on recent approaches to causal modeling developed in the machine-learning literature, which have extended Pearls do-calculus to incorporate cyclic causal interactions and multilevel causation. We also develop information-theoretic notions necessary to analyze causal information dynamics in our framework, introducing a causal generalization of the Partial Information Decomposition framework. We show how our causal framework helps to clarify conceptual issues in the contexts of complex trait analysis and cancer genetics, including assigning variation in an observed trait to genetic, epigenetic and environmental sources in the presence of epigenetic and environmental feedback processes, and variation in fitness to mutation processes in cancer using a multilevel causal model respectively, as well as relating causally-induced to observed variation in these variables via information theoretic bounds. In the process, we introduce a general class of multilevel causal evolutionary processes which connect evolutionary processes at multiple levels via coarse-graining relationships. Further, we show how a range of fitness models can be formulated in our framework, as well as a causal analog of Prices equation (generalizing the probabilistic Rice equation), clarifying the relationships between realized/probabilistic fitness and direct/indirect selection. Finally, we consider the potential relevance of our framework to foundational issues in biology and evolution, including supervenience, multilevel selection and individuality. Particularly, we argue that our class of multilevel causal evolutionary processes, in conjunction with a minimum description length principle, provides a conceptual framework in which identification of multiple levels of selection may be reduced to a model selection problem.more » « less
-
Students’ engagement with geographic information systems (GIS) can improve spatial skills, which are predictors of STEM success (Jant et al., 2019). We used a survey motivated by Eccles’s (2009) expectancy-value-cost framework to assess students’ perceptions of their computer science (CS) courses before and after participation in a GIS unit. The unit provided opportunities to apply GIS to inquiry-based projects focused on solving problems in their own communities. Across four teachers, 158 students participated in the GIS unit and completed the survey. We found that students’ reports of classroom equity predicted their expectancy for success in CS and their desire to take additional CS courses or major in CS. We also examined students’ performance on a geospatial problem-solving assessment to investigate their understanding of GIS and their spatial reasoning.more » « less
-
In this brief paper, we will share preliminary results of a study of how elementary-school teachers take up computational thinking (CT) ideas and incorporate them into their mathematics and science teaching. We describe the teachers’ school contexts, the professional development experiences in which they engaged, and our preliminary analyses of how they used computational thinking within their enacted lessons. In brief, the seven teachers in this study exhibited three patterns of implementation: (1) using computational thinking to guide their own planning and thinking; (2) using computational thinking to structure their lessons; and (3) presenting computational thinking concepts to students as general problem solving strategies.more » « less
-
Plug-and-play (PnP) prior is a well-known class of methods for solving imaging inverse problems by computing fixed-points of operators combining physical measurement models and learned image denoisers. While PnP methods have been extensively used for image recovery with known measurement operators, there is little work on PnP for solving blind inverse problems. We address this gap by presenting a new block-coordinate PnP (BC-PnP) method that efficiently solves this joint estimation problem by introducing learned denoisers as priors on both the unknown image and the unknown measurement operator. We present a new convergence theory for BC-PnP compatible with blind inverse problems by considering nonconvex data-fidelity terms and expansive denoisers. Our theory analyzes the convergence of BC-PnP to a stationary point of an implicit function associated with an approximate minimum mean-squared error (MMSE) denoiser. We numerically validate our method on two blind inverse problems: automatic coil sensitivity estimation in magnetic resonance imaging (MRI) and blind image deblurring. Our results show that BC-PnP provides an efficient and principled framework for using denoisers as PnP priors for jointly estimating measurement operators and images.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.3389/fpsyg.2024.1380178
