More Like this

1. Bayesian Optimization with Conformal Prediction Sets

   Stanton, S.; Maddox, W.; Wilson, A.G. (January 2023, Artificial Intelligence and Statistics)

   Bayesian optimization is a coherent, ubiquitous approach to decision-making under uncertainty, with applications including multi-arm bandits, active learning, and black-box optimization. Bayesian optimization selects decisions (i.e. objective function queries) with maximal expected utility with respect to the posterior distribution of a Bayesian model, which quantifies reducible, epistemic uncertainty about query outcomes. In practice, subjectively implausible outcomes can occur regularly for two reasons: 1) model misspecification and 2) covariate shift. Conformal prediction is an uncertainty quantification method with coverage guarantees even for misspecified models and a simple mechanism to correct for covariate shift. We propose conformal Bayesian optimization, which directs queries towards regions of search space where the model predictions have guaranteed validity, and investigate its behavior on a suite of black-box optimization tasks and tabular ranking tasks. In many cases we find that query coverage can be significantly improved without harming sample-efficiency. 

   more » « less
2. Design‐robust two‐way‐fixed‐effects regression for panel data

   https://doi.org/10.3982/QE1962  

   Arkhangelsky, Dmitry; Imbens, Guido W; Lei, Lihua; Luo, Xiaoman (November 2024, Quantitative Economics)

   We propose a new estimator for average causal effects of a binary treatment with panel data in settings with general treatment patterns. Our approach augments the popular two‐way‐fixed‐effects specification with unit‐specific weights that arise from a model for the assignment mechanism. We show how to construct these weights in various settings, including the staggered adoption setting, where units opt into the treatment sequentially but permanently. The resulting estimator converges to an average (over units and time) treatment effect under the correct specification of the assignment model, even if the fixed‐ effect model is misspecified. We show that our estimator is more robust than the conventional two‐way estimator: it remains consistent if either the assignment mechanism or the two‐way regression model is correctly specified. In addition, the proposed estimator performs better than the two‐way‐fixed‐effect estimator if the outcome model and assignment mechanism are locally misspecified. This strong robustness property underlines and quantifies the benefits of modeling the assignment process and motivates using our estimator in practice. We also discuss an extension of our estimator to handle dynamic treatment effects. 

   more » « less
3. Value Alignment Verification

   Brown, Daniel; Schneider, Jordan; Dragan, Anca; Niekum, Scott (January 2021, 38th International Conference on Machine Learning)

   As humans interact with autonomous agents to perform increasingly complicated, potentially risky tasks, it is important to be able to efficiently evaluate an agent’s performance and correctness. In this paper we formalize and theoretically analyze the problem of efficient value alignment verification: how to efficiently test whether the behavior of another agent is aligned with a human’s values. The goal is to construct a kind of “driver’s test” that a human can give to any agent which will verify value alignment via a minimal number of queries. We study alignment verification problems with both idealized humans that have an explicit reward function as well as problems where they have implicit values. We analyze verification of exact value alignment for rational agents and propose and analyze heuristic and approximate value alignment verification tests in a wide range of gridworlds and a continuous autonomous driving domain. Finally, we prove that there exist sufficient conditions such that we can verify exact and approximate alignment across an infinite set of test environments via a constant- query-complexity alignment test. 

   more » « less
4. Optimal Lending Contracts with Retrospective and Prospective Bias

   https://doi.org/10.1257/pandp.20231115  

   Bohren, J Aislinn; Hauser, Daniel N (May 2023, AEA Papers and Proceedings)

   Model misspecification is a common approach to model belief formation distortions. Misspecified models can be decomposed into two classes of distortions: prospective and retrospective biases (Bohren and Hauser 2023). Prospective biases correspond to distortions in forecasting future beliefs, while retrospective biases correspond to distortions in interpreting information ex post. We disentangle the impact of these two distortions on optimal lending contracts in the context of an entrepreneur who borrows to invest in a project. The entrepreneur learns about project quality from a signal, which she interprets with a misspecified model. A lender leverages each form of bias in distinct ways. 

   more » « less
5. Co-Designing with Users the Explanations for a Proactive Auto-Response Messaging Agent

   https://doi.org/10.1145/3604248  

   Jain, Pranut; Farzan, Rosta; Lee, Adam J. (September 2023, Proceedings of the ACM on Human-Computer Interaction)

   Explanations of AI Agents' actions are considered to be an important factor in improving users' trust in the decisions made by autonomous AI systems. However, as these autonomous systems evolve from reactive, i.e., acting on user input, to proactive, i.e., acting without requiring user intervention, there is a need to explore how the explanation for the actions of these agents should evolve. In this work, we explore the design of explanations through participatory design methods for a proactive auto-response messaging agent that can reduce perceived obligations and social pressure to respond quickly to incoming messages by providing unavailability-related context. We recruited 14 participants who worked in pairs during collaborative design sessions where they reasoned about the agent's design and actions. We qualitatively analyzed the data collected through these sessions and found that participants' reasoning about agent actions led them to speculate heavily on its design. These speculations significantly influenced participants' desire for explanations and the controls they sought to inform the agents' behavior. Our findings indicate a need to transform users' speculations into accurate mental models of agent design. Further, since the agent acts as a mediator in human-human communication, it is also necessary to account for social norms in its explanation design. Finally, user expertise in understanding their habits and behaviors allows the agent to learn from the user their preferences when justifying its actions. 

   more » « less