More Like this

1. Stronger Calibration Lower Bounds via Sidestepping

   Qiao, Mingda; Valiant, Gregory (January 2021, Proceedings of the Annual ACM Symposium on Theory of Computing)

   null (Ed.)

   We consider an online binary prediction setting where a forecaster observes a sequence of T bits one by one. Before each bit is revealed, the forecaster predicts the probability that the bit is 1. The forecaster is called well-calibrated if for each p in [0,1], among the n_p bits for which the forecaster predicts probability p, the actual number of ones, m_p, is indeed equal to p*n_p. The calibration error, defined as \sum_p |m_p - p n_p|, quantifies the extent to which the forecaster deviates from being well-calibrated. It has long been known that an O(T^(2/3)) calibration error is achievable even when the bits are chosen adversarially, and possibly based on the previous predictions. However, little is known on the lower bound side, except an sqrt(T) bound that follows from the trivial example of independent fair coin flips. In this paper, we prove an T^(0.528) bound on the calibration error, which is the first bound above the trivial sqrt(T) lowerbound for this setting. The technical contributions of our work include two lower bound techniques, early stopping and sidestepping, which circumvent the obstacles that have previously hindered strong calibration lower bounds. We also propose an abstraction of the prediction setting, termed the Sign-Preservation game, which may be of independent interest. This game has a much smaller state space than the full prediction setting and allows simpler analyses. The T^0.528 lower bound follows from a general reduction theorem that translates lower bounds on the game value of Sign-Preservation into lower bounds on the calibration error. 

   more » « less
2. Truthfulness of Decision-Theoretic Calibration Measures

   Qiao, Mingda; Zhao, Eric (June 2025, 38th Annual Conference on Learning Theory (COLT 2025))

   Calibration measures quantify how much a forecaster’s predictions violate calibration, which requires that forecasts are unbiased conditioning on the forecasted probabilities. Two important desiderata for a calibration measure are its decision-theoretic implications (i.e., downstream decision-makers that best respond to the forecasts are always no-regret) and its truthfulness (i.e., a forecaster approximately minimizes error by always reporting the true probabilities). Existing measures satisfy at most one of the properties, but not both. We introduce a new calibration measure termed subsampled step calibration, StepCEsub, that is both decision-theoretic and truthful. In particular, on any product distribution, StepCEsub is truthful up to an O(1) factor whereas prior decision-theoretic calibration measures suffer from an e−Ω(T)–Ω(T−−√) truthfulness gap. Moreover, in any smoothed setting where the conditional probability of each event is perturbed by a noise of magnitude c>0, StepCEsub is truthful up to an O(log(1/c)−−−−−−−√) factor, while prior decision-theoretic measures have an e−Ω(T)–Ω(T1/3) truthfulness gap. We also prove a general impossibility result for truthful decision-theoretic forecasting: any complete and decision-theoretic calibration measure must be discontinuous and non-truthful in the non-smoothed setting. 

   more » « less
3. Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

   https://doi.org/10.1186/s13071-022-05630-y  

   Holcomb, Karen M.; Mathis, Sarabeth; Staples, J. Erin; Fischer, Marc; Barker, Christopher M.; Beard, Charles B.; Nett, Randall J.; Keyel, Alexander C.; Marcantonio, Matteo; Childs, Marissa L.; et al (December 2023, Parasites & Vectors)

   Abstract Background West Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement. Methods We performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill. Results Simple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill. Conclusions Historical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases). Graphical Abstract 

   more » « less
4. Learning Residual Distributions with Diffusion Models for Probabilistic Wind Power Forecasting

   https://doi.org/10.3390/en18164226  

   Chen, Fuhao; Gao, Linyue (August 2025, Energies)

   Accurate and uncertainty-aware wind power forecasting is essential for reliable and cost-effective power system operations. This paper presents a novel probabilistic forecasting framework based on diffusion probabilistic models. We adopted a two-stage modeling strategy—a deterministic predictor first generates baseline forecasts, and a conditional diffusion model then learns the distribution of residual errors. Such a two-stage decoupling strategy improves learning efficiency and sharpens uncertainty estimation. We employed the elucidated diffusion model (EDM) to enable flexible noise control and enhance calibration, stability, and expressiveness. For the generative backbone, we introduced a time-series-specific diffusion Transformer (TimeDiT) that incorporates modular conditioning to separately fuse numerical weather prediction (NWP) inputs, noise, and temporal features. The proposed method was evaluated using the public database from ten wind farms in the Global Energy Forecasting Competition 2014 (GEFCom2014). We further compared our approach with two popular baseline models, i.e., a distribution parameter regression model and a generative adversarial network (GAN)-based model. Results showed that our method consistently achieves superior performance in both deterministic metrics and probabilistic accuracy, offering better forecast calibration and sharper distributions. 

   more » « less
5. Distributed Optimization over Time-Varying Networks: Imperfect Information with Feedback is as Good as Perfect Information

   https://doi.org/10.23919/ACC53348.2022.9867680  

   Reisizadeh, Hadi; Touri, Behrouz; Mohajer, Soheil (January 2022, 2022 American Control Conference (ACC))

   The convergence of an error-feedback algorithm is studied for decentralized stochastic gradient descent (DSGD) algorithm with compressed information sharing over time-varying graphs. It is shown that for both strongly-convex and convex cost functions, despite of imperfect information sharing, the convergence rates match those with perfect information sharing. To do so, we show that for strongly-convex loss functions, with a proper choice of a step-size, the state of each node converges to the global optimizer at the rate of O(T^{−1}). Similarly, for general convex cost functions, with a proper choice of step-size, we show that the value of loss function at a temporal average of each node’s estimates converges to the optimal value at the rate of O(T^{−1/2+ϵ }) for any ϵ > 0. 

   more » « less