More Like this

1. Surrogate Scoring Rules

   https://doi.org/10.1145/3565559  

   Liu, Yang; Wang, Juntao; Chen, Yiling (October 2022, ACM Transactions on Economics and Computation)

   Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this paper, we extend such scoring rules to settings where a principal elicits private probabilistic beliefs but only has access to agents’ reports. We name our solution Surrogate Scoring Rules (SSR). SSR is built on a bias correction step and an error rate estimation procedure for a reference answer defined using agents’ reports. We show that, with a little information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting with ground truth. As a by-product, SSR induce dominant uniform strategy truthfulness in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters. 

   more » « less
2. Surrogate Scoring Rules

   https://doi.org/10.1145/3391403.3399488  

   Liu, Yang; Wang, Juntao; Chen, Yiling (July 2020, ACM Conference on Economics and Computation)
3. Surrogate Model Selection for Design Space Approximation and Surrogate-based Optimization

   https://doi.org/10.1016/B978-0-12-818597-1.50056-4  

   Williams, Bianca; Cremaschi, Selen (July 2019, Computeraided Chemical Engineering)

   Surrogate models are used to map input data to output data when the actual relationship between the two is unknown or computationally expensive to evaluate for sensitivity analysis, uncertainty propagation and surrogate based optimization. This work evaluates the performance of eight surrogate modeling techniques for design space approximation and surrogate based optimization applications over a set of generated datasets with known characteristics. With this work, we aim to provide general rules for selecting an appropriate surrogate model form solely based on the characteristics of the data being modeled. The computational experiments revealed that, in general, multivariate adaptive regression spline models (MARS) and single hidden layer feed forward neural networks (ANN) yielded the most accurate predictions over the design space while Random Forest (RF) models most reliably identified the locations of the optimums when used for surrogate-based optimization. 

   more » « less
4. Deep Surrogate Assisted Generation of Environments

   Bhatt, Varun; Tjanaka, Bryon; Fontaine, Matthew; Nikolaidis, Stefanos (January 2022, Thirty-sixth Conference on Neural Information Processing Systems)
5. Surrogate Regret Bounds for Polyhedral Losses

   Frongillo, Rafael; Waggoner, Bo (January 2021, Advances in neural information processing systems)