More Like this

1. Calibration by Distribution Matching: Trainable Kernel Calibration Metrics

   Marx, Charles Thomas; Zalouk, Sofian; Ermon, Stefano (December 2023, Conference on Neural Information Processing Systems)
2. Interpolation Volume Calibration: A Multisensor Calibration Technique for Electromagnetic Trackers

   https://doi.org/10.1109/TRO.2012.2198929  

   Himberg, H.; Motai, Y.; Bradley, A. (October 2012, IEEE Transactions on Robotics)
3. Modular Conformal Calibration

   Marx, Charles; Zhao, Shengjia; Neiswanger, Willie; Ermon, Stefano (January 2022, International Conference on Machine Learning)
4. GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration

   Sung, Yoo Yeon; Fleisig, Eve; Hope, Yu; Upadhyay, Ishan; Boyd-Graber, Jordan (July 2025, Association for Computational Linguistics)

   As AI use becomes more common, it's important to measure not just whether the systems are correct but whether they know when they're incorrect. We propose a new metric to measure this mismatch between correctness and confidence, compare computer ability with human ability, and show that computers have a long way to go before they're well-calibrated. 

   more » « less
5. Truthfulness of Calibration Measures

   Haghtalab, Nika; Qiao, Mingda; Yang, Kunhe; Zhao, Eric (December 2024, Advances in Neural Information Processing Systems)

   We study calibration measures in a sequential prediction setup. In addition to rewarding accurate predictions (completeness) and penalizing incorrect ones (soundness), an important desideratum of calibration measures is truthfulness, a minimal condition for the forecaster not to be incentivized to exploit the system. Formally, a calibration measure is truthful if the forecaster (approximately) minimizes the expected penalty by predicting the conditional expectation of the next outcome, given the prior distribution of outcomes. We conduct a taxonomy of existing calibration measures. Perhaps surprisingly, all of them are far from being truthful. We introduce a new calibration measure termed the Subsampled Smooth Calibration Error (SSCE), which is complete and sound, and under which truthful prediction is optimal up to a constant multiplicative factor. In contrast, under existing calibration measures, there are simple distributions on which a polylogarithmic (or even zero) penalty is achievable, while truthful prediction leads to a polynomial penalty. 

   more » « less