An official website of the United States government
We consider the crowdsourcing setting where, in response to the assigned tasks, agents strategically decide both how much effort to exert (from a continuum) and whether to manipulate their reports. The goal is to design payment mechanisms that (1) satisfy limited liability (all payments are non-negative), (2) reduce the principal’s cost of budget, (3) incentivize effort and (4) incentivize truthful responses. In our framework, the payment mechanism composes a performance measurement, which noisily evaluates agents’ effort based on their reports, and a payment function, which converts the scores output by the performance measurement to payments. Previous literature suggests applying a peer prediction mechanism combined with a linear payment function. This method can achieve either (1), (3) and (4), or (2), (3) and (4) in the binary effort setting. In this paper, we suggest using a rank-order payment function (tournament). Assuming Gaussian noise, we analytically optimize the rank-order payment function, and identify a sufficient statistic, sensitivity, which serves as a metric for optimizing the performance measurements. This helps us obtain (1), (2) and (3) simultaneously. Additionally, we show that adding noise to agents’ scores can preserve the truthfulness of the performance measurements under the non-linear tournament, which gives us all four objectives. Our real-data estimated agent-based model experiments show that our method can greatly reduce the payment of effort elicitation while preserving the truthfulness of the performance measurement. In addition, we empirically evaluate several commonly used performance measurements in terms of their sensitivities and strategic robustness.
more »
« less
Asymmetric shocks in contests: Theory and experiment
Under optimal tournament design, equilibrium effort is invariant to the shape of the mean-zero additive stochastic component, often referred to as a “shock” or “noise”. We report data from laboratory experiments providing the first test of this prediction. Consistent with theory, we find that average effort does not significantly differ between a negatively skewed and uniform shock distribution. In addition, we test a second theoretical prediction that, in winner tournaments, when the shock distribution is asymmetric as in our design, one should exert minimum effort whenever one’s competitors are exerting above equilibrium effort. With a symmetric shock distribution as in our design, efforts should generally remain substantial, even when one’s competitors are exerting effort above equilibrium value. Our data reveal that subjects actively engage in the tournament even when faced with aggressive competitors under both shock distributions.
more »
« less
- Award ID(s):
- 2048519
- PAR ID:
- 10539255
- Publisher / Repository:
- Elsevier
- Date Published:
- Journal Name:
- Journal of Economic Behavior and Organization
- Volume:
- 216
- ISSN:
- 0167-2681
- Page Range / eLocation ID:
- 243-267
- Subject(s) / Keyword(s):
- Asymmetric random shock Tournament Laboratory experiment Off-equilibrium behavior
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Though hackathons are successful in attracting large crowds, they may not be sufficiently effective for broadening participation in computing, because they lack appeal for underrepresented groups in computing, and for people with family and job obligations. We propose a contrasting model for creating interest in computing, by making coding a spectator sport. We present an experience report on the design and implementation of a coding tournament, including survey results that informed the design of the system along with post-event questionnaire data from participants, exploring their attitudes towards different coding events. We find that coding tournaments can be an effective and engaging alternative to hackathons, and that they can motivate some audience members to pursue more coding activities, and possibly even participate as competitors in future tournaments.more » « less
-
We initiate the study of information elicitation mechanisms for a crowd containing both self-interested agents, who respond to incentives, and adversarial agents, who may collude to disrupt the system. Our mechanisms work in the peer prediction setting where ground truth need not be accessible to the mechanism or even exist. We provide a meta-mechanism that reduces the design of peer prediction mechanisms to a related robust learning problem. The resulting mechanisms are ϵ-informed truthful, which means truth-telling is the highest paid ϵ-Bayesian Nash equilibrium (up to ϵ-error) and pays strictly more than uninformative equilibria. The value of ϵ depends on the properties of robust learning algorithm, and typically limits to 0 as the number of tasks and agents increase. We show how to use our meta-mechanism to design mechanisms with provable guarantees in two important crowdsourcing settings even when some agents are self-interested and others are adversarial.more » « less
-
The Random Forests classifier, a widely utilized off-the-shelf classification tool, assumes training and test samples come from the same distribution as other standard classifiers. However, in safety-critical scenarios like medical diagnosis and network attack detection, discrepancies between the training and test sets, including the potential presence of novel outlier samples not appearing during training, can pose significant challenges. To address this problem, we introduce the Conformalized Semi-Supervised Random Forest (CSForest), which couples the conformalization technique Jackknife+aB with semi-supervised tree ensembles to construct a set-valued prediction 𝐶(𝑥). Instead of optimizing over the training distribution, CSForest employs unlabeled test samples to enhance accuracy and flag unseen outliers by generating an empty set. Theoretically, we establish CSForest to cover true labels for previously observed inlier classes under arbitrarily label-shift in the test data. We compare CSForest with state-of-the-art methods using synthetic examples and various real-world datasets, under different types of distribution changes in the test domain. Our results highlight CSForest’s effective prediction of inliers and its ability to detect outlier samples unique to the test data. In addition, CSForest shows persistently good performance as the sizes of the training and test sets vary. Codes of CSForest are available at https://github.com/yujinhan98/CSForest.more » « less
-
null (Ed.)Monocular depth predictors are typically trained on large-scale training sets which are naturally biased w.r.t the distribution of camera poses. As a result, trained predic- tors fail to make reliable depth predictions for testing exam- ples captured under uncommon camera poses. To address this issue, we propose two novel techniques that exploit the camera pose during training and prediction. First, we in- troduce a simple perspective-aware data augmentation that synthesizes new training examples with more diverse views by perturbing the existing ones in a geometrically consis- tent manner. Second, we propose a conditional model that exploits the per-image camera pose as prior knowledge by encoding it as a part of the input. We show that jointly ap- plying the two methods improves depth prediction on im- ages captured under uncommon and even never-before-seen camera poses. We show that our methods improve perfor- mance when applied to a range of different predictor ar- chitectures. Lastly, we show that explicitly encoding the camera pose distribution improves the generalization per- formance of a synthetically trained depth predictor when evaluated on real images.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- The DOI is not currently available.
