An official website of the United States government
Counterfactual estimators enable the use of existing log data to estimate how some new target recommendation policy would have performed, if it had been used instead of the policy that logged the data. We say that those estimators work ”off-policy”, since the policy that logged the data is different from the target policy. In this way, counterfactual estimators enable Off-policy Evaluation (OPE) akin to an unbiased offline A/B test, as well as learning new recommendation policies through Off-policy Learning (OPL). The goal of this tutorial is to summarize Foundations, Implementations, and Recent Advances of OPE/OPL. Specifically, we will introduce the fundamentals of OPE/OPL and provide theoretical and empirical comparisons of conventional methods. Then, we will cover emerging practical challenges such as how to take into account combinatorial actions, distributional shift, fairness of exposure, and two-sided market structures. We will then present Open Bandit Pipeline, an open-source package for OPE/OPL, and how it can be used for both research and practical purposes. We will conclude the tutorial by presenting real-world case studies and future directions.
more »
« less
Evaluating a PI Policy with Data
At our organizations, policies are often implemented then never reviewed again. We often don’t know why the policy existed in the first place. In this session, we will discuss how the data we already collect can be used to evaluate existing policies, proposed policies, and potential changes. We will walk through an example of how a proposed institutional policy was examined through the use of data on research proposals and awards, and answer the question “Do we really need this policy?” Bring your examples to talk through options for policy evaluation! Presented at the 2024 Research Analytics Summit in Albuquerque, NM
more »
« less
- Award ID(s):
- 2324388
- PAR ID:
- 10566922
- Publisher / Repository:
- University of Kentucky Libraries
- Date Published:
- Subject(s) / Keyword(s):
- FOS: Computer and information sciences
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
If you want great research analytics, you’re going to need great data governance for your research metadata. That’s a tall order when the best information often spans incompatible systems, departments, policies, and mindsets. Research metadata governance is a great illustration of how research analytics is not just a technology problem. It happens at the intersection of technology, policy, process, and culture. Do any of us feel like our organization is “nailing” metadata governance? What level of research analytics could you achieve if there could be more alignment and cooperation around this kind of governance? In this session, you will join your fellow attendees in a facilitated design thinking workshop around the culture and organizational patterns of research metadata governance. We will explore the problem space and the solution space together. And we’ll highlight the group’s best findings. Whether you’re at the technical implementation level or the VP level, you will have seen and experienced good and bad patterns of how research metadata is being governed at your institution. These experiences are valuable to your fellow community members. Using design thinking, we will explore the collective intelligence on this complex topic, shoulder to shoulder. You’ll leave with a better understanding of how the patterns and struggles of data governance at your institution correlate with those of others. You’ll gain new ideas on how to address the hard cultural and organizational problems of governance. And you’ll meet new colleagues who you can stay connected with. Finally, you’ll gain an appreciation of how design thinking techniques themselves can be useful for these types of challenges at your own organization, and how they help to create understanding and alignment.more » « less
-
The public acknowledges the importance of water quality, and threats to water quality can provoke strong emotional responses. Despite this, the public often resists policies protecting water quality. Research with 349 US residents demonstrated that (1) emotions about specific water policies were more predictive of policy support than emotions about water quality and (2) hope about water policies was a particularly strong predictor of water policy support. In both between-person and within-person analyses, water-policy hope was a stronger predictor of water-policy support than water-policy anxiety, anger, and neutral affect–although these other emotions were related to water-policy support. These findings among water-policy emotions replicated results from a Pilot study with 148 US undergraduate students. The main study also demonstrated that water-policy support increased when policy descriptions explained how policies would improve water quality via hydro systems, and it did so by increasing feelings of water-policy hope. This research suggests that a full range of affective reactions to water policy and water quality should be considered when motivating support for policies protecting water quality.more » « less
-
Abstract Despite the reality that advocates frequently expend significant resources to pass symbolic policies, this policy design has often been neglected by policy studies scholarship. We combine policy design and policy feedback theory to examine this oft overlooked policy design in practice using the case of California's human right to water law (Assembly Bill 685, or AB 685). Through semi‐structured interviews, archival research, and document analysis, we reveal how grassroots advocates deliberately and effectively pursued AB 685 to build power across the water justice movement and catalyze narrative change about drinking water access, while also building state responsiveness on the topic. These interpretive policy feedback effects then accelerated the policy's resource effects through formal policy changes in funding allocations, administrative structures, and regulatory systems. Collectively, feedbacks from AB 685 have transformed the sociopolitics of drinking water access. Contrary to prevailing wisdom, the policy's ambiguity proved key to building the broad coalition necessary to accomplish these changes, and it facilitated work across policy venues and governance scales through time, which is critical to enacting transformational change. Based on these findings, we argue that symbolic policies merit attention as a potentially advantageous policy design for social movements seeking social change and transformation.more » « less
-
Control of networked systems, comprised of interacting agents, is often achieved through modeling the underlying interactions. Constructing accurate models of such interactions–in the meantime–can become prohibitive in applications. Data-driven control methods avoid such complications by directly synthesizing a controller from the observed data. In this paper, we propose an algorithm referred to as Data-driven Structured Policy Iteration (D2SPI), for synthesizing an efficient feedback mechanism that respects the sparsity pattern induced by the underlying interaction network. In particular, our algorithm uses temporary “auxiliary” communication links in order to enable the required information exchange on a (smaller) sub-network during the “learning phase”—links that will be removed subsequently for the final distributed feedback synthesis. We then proceed to show that the learned policy results in a stabilizing structured policy for the entire network. Our analysis is then followed by showing the stability and convergence of the proposed distributed policies throughout the learning phase, exploiting a construct referred to as the “Patterned monoid.” The performance of D2SPI is then demonstrated using representative simulation scenarios.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Proceeding:
- https://doi.org/10.13023/XE5P-SC13
