An official website of the United States government
Strictly proper scoring rules (SPSR) are incentive compatible for eliciting information about random variables from strategic agents when the principal can reward agents after the realization of the random variables. They also quantify the quality of elicited information, with more accurate predictions receiving higher scores in expectation. In this paper, we extend such scoring rules to settings where a principal elicits private probabilistic beliefs but only has access to agents’ reports. We name our solution Surrogate Scoring Rules (SSR). SSR is built on a bias correction step and an error rate estimation procedure for a reference answer defined using agents’ reports. We show that, with a little information about the prior distribution of the random variables, SSR in a multi-task setting recover SPSR in expectation, as if having access to the ground truth. Therefore, a salient feature of SSR is that they quantify the quality of information despite the lack of ground truth, just as SPSR do for the setting with ground truth. As a by-product, SSR induce dominant uniform strategy truthfulness in reporting. Our method is verified both theoretically and empirically using data collected from real human forecasters.
more »
« less
Trajectory-Oriented Optimization of Stochastic Epidemiological Models
Epidemiological models must be calibrated to ground truth for downstream tasks such as producing forward projections or running what-if scenarios. The meaning of calibration changes in case of a stochastic model since output from such a model is generally described via an ensemble or a distribution. Each member of the ensemble is usually mapped to a random number seed (explicitly or implicitly). With the goal of finding not only the input parameter settings but also the random seeds that are consistent with the ground truth, we propose a class of Gaussian process (GP) surrogates along with an optimization strategy based on Thompson sampling. This Trajectory Oriented Optimization (TOO) approach produces actual trajectories close to the empirical observations instead of a set of parameter settings where only the mean simulation behavior matches with the ground truth.
more »
« less
- Award ID(s):
- 2200234
- PAR ID:
- 10497716
- Publisher / Repository:
- IEEE
- Date Published:
- ISBN:
- 979-8-3503-6966-3
- Page Range / eLocation ID:
- 1244 to 1255
- Format(s):
- Medium: X
- Location:
- San Antonio, TX, USA
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
null (Ed.)Modelling of cardiac electrical behaviour has led to important mechanistic insights, but important challenges, including uncertainty in model formulations and parameter values, make it difficult to obtain quantitatively accurate results. An alternative approach is combining models with observations from experiments to produce a data-informed reconstruction of system states over time. Here, we extend our earlier data-assimilation studies using an ensemble Kalman filter to reconstruct a three-dimensional time series of states with complex spatio-temporal dynamics using only surface observations of voltage. We consider the effects of several algorithmic and model parameters on the accuracy of reconstructions of known scroll-wave truth states using synthetic observations. In particular, we study the algorithm’s sensitivity to parameters governing different parts of the process and its robustness to several model-error conditions. We find that the algorithm can achieve an acceptable level of error in many cases, with the weakest performance occurring for model-error cases and more extreme parameter regimes with more complex dynamics. Analysis of the poorest-performing cases indicates an initial decrease in error followed by an increase when the ensemble spread is reduced. Our results suggest avenues for further improvement through increasing ensemble spread by incorporating additive inflation or using a parameter or multi-model ensemble. This article is part of the theme issue ‘Uncertainty quantification in cardiac and cardiovascular modelling and simulation’.more » « less
-
Training deep learning models for unbiased stereology requires a large data set with associated ground truth. However manual ground truth annotation is tedious, time-consuming, and expert dependent. We propose an active deep learning method for automatic stereology counts using a snapshot ensemble approach. The method provides a confidence score for each mask in an unlabeled pool that reduces user verification to only images with high information content for training the deep learning model. The proposed method reduces the error rate to less than 1% for unbiased stereology cell counts on immunostained brain cells compared to manual stereology and requires ~25% less expert verification time compared to a previously proposed iterative deep learning approach.more » « less
-
Tracking activities holds great potential to improve the well-being of older adults, yet the accuracy of activity trackers for this demographic remains in question. Evaluating this accuracy requires ground-truth data directly from older adults, which has largely been gathered in controlled laboratory settings or labeled by researchers. Moreover, considering the diversity in older adults' activity engagement and tracking preferences, personalized activity tracking appears necessary. We demonstrate that older adults can benefit from personalized activity trackers by showing that cadence thresholds for stepping intensities vary within this group. However, collecting ground-truth data from older adults in real-world settings poses unique challenges. This paper examines two sources of ground-truth labels for the smartwatch Inertial Measurement Unit (IMU) data collected with older adults. Using verbal self-reports and a thigh-worn activity tracker, we assess their viability as ground-truth sources in natural settings. Additionally, we evaluate the costs and benefits of triangulating these sources as a ground-truth proxy. Our findings reveal two main costs: data shrinkage and notable effort from both contributors and data stewards. Simultaneously, we observe improved data quality and a greater ability to identify error sources when evaluating a trained model.more » « less
-
Consider a network of agents that all want to guess the correct value of some ground truth state. In a sequential order, each agent makes its decision using a single private signal which has a constant probability of error, as well as observations of actions from its network neighbors earlier in the order. We are interested in enabling network-wide asymptotic truth learning – that in a network of n agents, almost all agents make a correct prediction with probability approaching one as n goes to infinity. In this paper we study both random orderings and carefully crafted decision orders with respect to the graph topology as well as sufficient or necessary conditions for a graph to support such a good ordering. We first show that on a sparse graph of average constant degree with a random ordering asymptotic truth learning does not happen. We then show a rather modest sufficient condition to enable asymptotic truth learning. With the help of this condition we characterize graphs generated from the Erdös Rényi model and preferential attachment model. In an Erdös Rényi graph, unless the graph is super sparse (with O(n) edges) or super dense (nearly a complete graph), there exists a decision ordering that supports asymptotic truth learning. Similarly, any preferential attachment network with a constant number of edges per node can achieve asymptotic truth learning under a carefully designed ordering but not under either a random ordering nor the arrival order. We also evaluated a variant of the decision ordering on different network topologies and demonstrated clear effectiveness in improving truth learning over random orderings.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/WSC60868.2023.10408258
