An official website of the United States government
In many real-world problems, we are faced with the problem of selecting the best among a finite number of alternatives, where the best alternative is determined based on context specific information. In this work, we study the contextual Ranking and Selection problem under a finite-alternative-finite-context setting, where we aim to find the best alternative for each context. We use a separate Gaussian process to model the reward for each alternative and derive the large deviations rate function for both the expected and worst-case contextual probability of correct selection. We propose the GP-C-OCBA sampling policy, which uses the Gaussian process posterior to iteratively allocate observations to maximize the rate function. We prove its consistency and show that it achieves the optimal convergence rate under the assumption of a non-informative prior. Numerical experiments show that our algorithm is highly competitive in terms of sampling efficiency, while having significantly smaller computational overhead.
more »
« less
This content will become publicly available on November 1, 2026
Selection of the Most Probable Best
In practice, simulation parameters are estimated from finite data, which introduces model uncertainty. In “Selection of the Most Probable Best,” Taeho Kim, Kyoung-Kuk Kim, and Eunhye Song propose a new estimator of the optimum, the most probable best (MPB), for ranking and selection (R&S) when the model uncertainty is characterized by a posterior distribution on the parameter given data. Defined as the solution with the largest posterior probability of optimality, selecting the MPB requires simulating at all parameter-solution combinations. The authors focus on when the posterior has a finite support and derive the convergence rate of the probability of correctly selecting the MPB as a function of sampling fractions of all solution-parameter pairs. They propose a lower bound on the rate function that allows easier characterization of the optimal sampling fractions and devise a sequential sampling algorithm from it. The algorithm shows superior empirical performance over benchmarks.
more »
« less
- PAR ID:
- 10704905
- Publisher / Repository:
- INFORMS
- Date Published:
- Journal Name:
- Operations Research
- Volume:
- 73
- Issue:
- 6
- ISSN:
- 0030-364X
- Page Range / eLocation ID:
- 3199 to 3218
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
How can we collect the most useful labels to learn a model selection policy, when presented with arbitrary heterogeneous data streams? In this paper, we formulate this task as a contextual active model selection problem, where at each round the learner receives an unlabeled data point along with a context. The goal is to output the best model for any given context without obtaining an excessive amount of labels. In particular, we focus on the task of selecting pre-trained classifiers, and propose a contextual active model selection algorithm (CAMS), which relies on a novel uncertainty sampling query criterion defined on a given policy class for adaptive model selection. In comparison to prior art, our algorithm does not assume a globally optimal model. We provide rigorous theoretical analysis for the regret and query complexity under both adversarial and stochastic settings. Our experiments on several benchmark classification datasets demonstrate the algorithm’s effectiveness in terms of both regret and query complexity. Notably, to achieve the same accuracy, CAMS incurs less than 10% of the label cost when compared to the best online model selection baselines on CIFAR10.more » « less
-
Data-driven Deep Learning (DL) models have revolutionized autonomous systems, but ensuring their safety and reliability necessitates the assessment of predictive confidence or uncertainty. Bayesian DL provides a principled approach to quantify uncertainty via probability density functions defined over model parameters. However, the exact solution is intractable for most DL models, and the approximation methods, often based on heuristics, suffer from scalability issues and stringent distribution assumptions and may lack theoretical guarantees. This work develops a Sequential Importance Sampling framework that approximates the posterior probability density function through weighted samples (or particles), which can be used to find the mean, variance, or higher-order moments of the posterior distribution. We demonstrate that propagating particles, which capture information about the higher-order moments, through the layers of the DL model results in increased robustness to natural and malicious noise (adversarial attacks). The variance computed from these particles effectively quantifies the model’s decision uncertainty, demonstrating well-calibrated and accurate predictive confidence.more » « less
-
We propose an empirical Bayes formulation of the structure learning problem, where the prior specification assumes that all node variables have the same error variance, an assumption known to ensure the identifiability of the underlying causal directed acyclic graph. To facilitate efficient posterior computation, we approximate the posterior probability of each ordering by that of a best directed acyclic graph model, which naturally leads to an order-based Markov chain Monte Carlo algorithm. Strong selection consistency for our model in high-dimensional settings is proved under a condition that allows heterogeneous error variances, and the mixing behaviour of our sampler is theoretically investigated. Furthermore, we propose a new iterative top-down algorithm, which quickly yields an approximate solution to the structure learning problem and can be used to initialize the Markov chain Monte Carlo sampler. We demonstrate that our method outperforms other state-of-the-art algorithms under various simulation settings, and conclude the paper with a single-cell real-data study illustrating practical advantages of the proposed method.more » « less
-
Abstract Bayesian statistics has gained popularity in psychological research due to its intuitive uncertainty quantification and convenient information‐updating rules. In many applications, however, prior distributions are introduced merely as instruments to facilitate computation, rather than as representations of genuine subjective belief. Consequently, relying on standard Bayesian justifications for inferential procedures becomes conceptually ungrounded. In this paper, we recommend evaluating finite‐sample performance over repeated sampling of data and parameters as an alternative justification for “pragmatic Bayes.” We demonstrate a key vulnerability in the usual posterior‐based inference: when analysts' chosen prior distribution mismatches the true parameter‐generating process, Bayesian inference can be misleading in the long run. Given that this true process is rarely known in practice, we propose a safer alternative: calibrating Bayesian credible regions to achieve frequentist validity. This latter criterion is stronger and guarantees validity of Bayesian inference regardless of the underlying parameter‐generating mechanism. To solve the calibration problem in practice, we propose a novel stochastic approximation algorithm. Monte Carlo experiments are conducted and reported, in which we observe that uncalibrated Bayesian inference can be liberal under certain parameter‐generating scenarios, whereas our calibrated solution consistently maintain validity. We also illustrate the proposed calibration procedure using a real‐data example involving location‐scale regression.more » « less
