Search for: All records

Current practice for evaluating recommender systems typically focuses on point estimates of user-oriented effectiveness metrics or business metrics, sometimes combined with additional metrics for considerations such as diversity and novelty. In this paper, we argue for the need for researchers and practitioners to attend more closely to various distributions that arise from a recommender system (or other information access system) and the sources of uncertainty that lead to these distributions. One immediate implication of our argument is that both researchers and practitioners must report and examine more thoroughly the distribution of utility between and within different stakeholder groups. However, distributions of various forms arise in many more aspects of the recommender systems experimental process, and distributional thinking has substantial ramifications for how we design, evaluate, and present recommender systems evaluation and research results. Leveraging and emphasizing distributions in the evaluation of recommender systems is a necessary step to ensure that the systems provide appropriate and equitably-distributed benefit to the people they affect. 

There is increasing attention to evaluating the fairness of search system ranking decisions. These metrics often consider the membership of items to particular groups, often identified using protected attributes such as gender or ethnicity. To date, these metrics typically assume the availability and completeness of protected attribute labels of items. However, the protected attributes of individuals are rarely present, limiting the application of fair ranking metrics in large scale systems. In order to address this problem, we propose a sampling strategy and estimation technique for four fair ranking metrics. We formulate a robust and unbiased estimator which can operate even with very limited number of labeled items. We evaluate our approach using both simulated and real world data. Our experimental results demonstrate that our method can estimate this family of fair ranking metrics and provides a robust, reliable alternative to exhaustive or random data annotation. 

We introduce the concept of \emph{expected exposure} as the average attention ranked items receive from users over repeated samples of the same query. Furthermore, we advocate for the adoption of the principle of equal expected exposure: given a fixed information need, no item should receive more or less expected exposure than any other item of the same relevance grade. We argue that this principle is desirable for many retrieval objectives and scenarios, including topical diversity and fair ranking. Leveraging user models from existing retrieval metrics, we propose a general evaluation methodology based on expected exposure and draw connections to related metrics in information retrieval evaluation. Importantly, this methodology relaxes classic information retrieval assumptions, allowing a system, in response to a query, to produce a \emph{distribution over rankings} instead of a single fixed ranking. We study the behavior of the expected exposure metric and stochastic rankers across a variety of information access conditions, including \emph{ad hoc} retrieval and recommendation. We believe that measuring and optimizing expected exposure metrics using randomization opens a new area for retrieval algorithm development and progress. 

Although environmental variability and predictability have been proposed as the underlying ecological context in which transgenerational plasticity (TGP) arises, the adaptive significance and interaction with within‐generation plasticity (WGP) in such scenarios is still poorly understood. To investigate these questions, we considered the tolerance to upper thermal limits of larvae and adults of the desert endemicDrosophila mojavensisadapted to different climatic regions (Desert vs. Mediterranean climate).

Thermal plasticity was investigated by acclimating parents and offspring at 36°C (vs. at 25°C). We then used historical temperature variation data from both regions to perform individual‐based simulations by modelling expected components of adaptive plasticity in multiple life stages.

Our results indicated that thermal response to ramping heat shocks was more pronounced in larvae, where acclimation treatments in parents and offspring increased their heat‐shock performance, while heat knockdown in adults was only increased by offspring acclimation of adults. The relative contribution ofWGPandTGPwas greater for the population from the more thermally variable Sonoran Desert.

Similarly, individual‐based simulations of evolving maternal effects indicated that variation in tolerance to upper thermal limits across life stages and climates is expected from its adaptive significance in response to environmental predictability.

Our approach offers a new perspective and interpretation of adaptive plasticity, demonstrating that environmental predictability can drive thermal responses across generations and life stages in a scenario with regional climate variability.

A freePlain Language Summarycan be found within the Supporting Information of this article.