skip to main content
US FlagAn official website of the United States government
dot gov icon
Official websites use .gov
A .gov website belongs to an official government organization in the United States.
https lock icon
Secure .gov websites use HTTPS
A lock ( lock ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.


Title: Knowing what to know: Implications of the choice of prior distribution on the behavior of adaptive design optimization
Abstract Adaptive design optimization (ADO) is a state-of-the-art technique for experimental design (Cavagnaro et al., 2010). ADO dynamically identifies stimuli that, in expectation, yield the most information about a hypothetical construct of interest (e.g., parameters of a cognitive model). To calculate this expectation, ADO leverages the modeler’s existing knowledge, specified in the form of a prior distribution.Informativepriors align with the distribution of the focal construct in the participant population. This alignment is assumed by ADO’s internal assessment of expected information gain. If the prior is insteadmisinformative, i.e., does not align with the participant population, ADO’s estimates of expected information gain could be inaccurate. In many cases, the true distribution that characterizes the participant population is unknown, and experimenters rely on heuristics in their choice of prior and without an understanding of how this choice affects ADO’s behavior. Our work introduces a mathematical framework that facilitates investigation of the consequences of the choice of prior distribution on the efficiency of experiments designed using ADO. Through theoretical and empirical results, we show that, in the context ofprior misinformation, measures of expected information gain are distinct from the correctness of the corresponding inference. Through a series of simulation experiments, we show that, in the case of parameter estimation, ADO nevertheless outperforms other design methods. Conversely, in the case of model selection, misinformative priors can lead inference to favor the wrong model, and rather than mitigating this pitfall, ADO exacerbates it.  more » « less
Award ID(s):
2049896
PAR ID:
10521893
Author(s) / Creator(s):
; ;
Publisher / Repository:
Springer Science + Business Media
Date Published:
Journal Name:
Behavior Research Methods
Volume:
56
Issue:
7
ISSN:
1554-3528
Format(s):
Medium: X Size: p. 7102-7125
Size(s):
p. 7102-7125
Sponsoring Org:
National Science Foundation
More Like this
  1. Spiegler, Ran (Ed.)
    We introduce a model of random ambiguity aversion. Choice is stochastic due to unobserved shocks to both information and ambiguity aversion. This is modeled as a random set of beliefs in the maxmin expected utility model of Gilboa and Schmeidler (1989). We characterize the model and show that the distribution of ambiguity aversion can be uniquely identified using binary choices. A novel stochastic order on random sets is introduced that characterizes greater uncertainty aversion under stochastic choice. If the set of priors is the Aumann expectation of the random set, then choices satisfy dynamic consistency. This corresponds to an agent who knows the distribution of signals but is uncertain about how to interpret signal realizations. More broadly, the analysis of stochastic properties of random ambiguity attitudes provides a theoretical foundation for the study of models of random non-linear utility. 
    more » « less
  2. Despite many successful examples in which probabilistic inference can account for perception, we have little understanding of how the brain represents and uses structured priors that capture the complexity of natural input statistics. Here we construct a recurrent circuit model that can implicitly represent priors over latent variables, and combine them with sensory and contextual sources of information to encode task-specific posteriors. Inspired by the recent success of diffusion models as means of learning and using priors over images, our model uses dendritic nonlinearities optimized for denoising, and stochastic somatic integration with the degree of noise modulated by an oscillating global signal. Combining these elements into a recurrent network yields a stochastic dynamical system that samples from the prior at a rate prescribed by the period of the global oscillator. Additional inputs reflecting sensory or top-down contextual information alter these dynamics to generate samples from the corresponding posterior, with different input gating patterns selecting different inference tasks. We demonstrate that this architecture can sample from low dimensional nonlinear manifolds and multimodal posteriors. Overall, the model provides a new framework for circuit-level representation of probabilistic information, in a format that facilitates flexible inference. 
    more » « less
  3. null (Ed.)
    Abstract Designers make information acquisition decisions, such as where to search and when to stop the search. Such decisions are typically made sequentially, such that at every search step designers gain information by learning about the design space. However, when designers begin acquiring information, their decisions are primarily based on their prior knowledge. Prior knowledge influences the initial set of assumptions that designers use to learn about the design space. These assumptions are collectively termed as inductive biases. Identifying such biases can help us better understand how designers use their prior knowledge to solve problems in the light of uncertainty. Thus, in this study, we identify inductive biases in humans in sequential information acquisition tasks. To do so, we analyze experimental data from a set of behavioral experiments conducted in the past [1–5]. All of these experiments were designed to study various factors that influence sequential information acquisition behaviors. Across these studies, we identify similar decision making behaviors in the participants in their very first decision to “choose x”. We find that their choices of “x” are not uniformly distributed in the design space. Since such experiments are abstractions of real design scenarios, it implies that further contextualization of such experiments would only increase the influence of these biases. Thus, we highlight the need to study the influence of such biases to better understand designer behaviors. We conclude that in the context of Bayesian modeling of designers’ behaviors, utilizing the identified inductive biases would enable us to better model designer’s priors for design search contexts as compared to using non-informative priors. 
    more » « less
  4. Abstract The identification and description of point sources is one of the oldest problems in astronomy, yet even today the correct statistical treatment for point sources remains one of the field’s hardest problems. For dim or crowded sources, likelihood-based inference methods are required to estimate the uncertainty on the characteristics of the source population. In this work, a new parametric likelihood is constructed for this problem using compound Poisson generator (CPG) functionals that incorporate instrumental effects from first principles. We demonstrate that the CPG approach exhibits a number of advantages over non-Poissonian template fitting (NPTF)—an existing method—in a series of test scenarios in the context of X-ray astronomy. These demonstrations show that the effect of the point-spread function, effective area, and choice of point-source spatial distribution cannot, generally, be factorized as they are in NPTF, while the new CPG construction is validated in these scenarios. Separately, an examination of the diffuse-flux emission limit is used to show that most simple choices of priors on the standard parameterization of the population model can result in unexpected biases: when a model comprising both a point-source population and diffuse component is applied to this limit, nearly all observed flux will be assigned to either the population or to the diffuse component. A new parameterization is presented for these priors that properly estimates the uncertainties in this limit. In this choice of priors, CPG correctly identifies that the fraction of flux assigned to the population model cannot be constrained by the data. 
    more » « less
  5. Abstract We presentnimbus: a hierarchical Bayesian framework to infer the intrinsic luminosity parameters of kilonovae (KNe) associated with gravitational-wave (GW) events, based purely on nondetections. This framework makes use of GW 3D distance information and electromagnetic upper limits from multiple surveys for multiple events and self-consistently accounts for the finite sky coverage and probability of astrophysical origin. The framework is agnostic to the brightness evolution assumed and can account for multiple electromagnetic passbands simultaneously. Our analyses highlight the importance of accounting for model selection effects, especially in the context of nondetections. We show our methodology using a simple, two-parameter linear brightness model, taking the follow-up of GW190425 with the Zwicky Transient Facility as a single-event test case for two different prior choices of model parameters: (i) uniform/uninformative priors and (ii) astrophysical priors based on surrogate models of Monte Carlo radiative-transfer simulations of KNe. We present results under the assumption that the KN is within the searched region to demonstrate functionality and the importance of prior choice. Our results show consistency withsimsurvey—an astronomical survey simulation tool used previously in the literature to constrain the population of KNe. While our results based on uniform priors strongly constrain the parameter space, those based on astrophysical priors are largely uninformative, highlighting the need for deeper constraints. Future studies with multiple events having electromagnetic follow-up from multiple surveys should make it possible to constrain the KN population further. 
    more » « less