More Like this

1. Scalable Gaussian Process Inference with Finite-data Mean and Variance Guarantees

   Huggins, Jonathan H.; Campbell, Trevor; Kasprzak, Mikolaj; Broderick, Tamara (April 2019, Proceedings of Machine Learning Research)

   Gaussian processes (GPs) offer a flexible class of priors for nonparametric Bayesian regression, but popular GP posterior inference methods are typically prohibitively slow or lack desirable finite-data guarantees on quality. We develop a scalable approach to approximate GP regression, with finite-data guarantees on the accuracy of our pointwise posterior mean and variance estimates. Our main contribution is a novel objective for approximate inference in the nonparametric setting: the preconditioned Fisher (pF) divergence. We show that unlike the Kullback–Leibler divergence (used in variational inference), the pF divergence bounds bounds the 2-Wasserstein distance, which in turn provides tight bounds on the pointwise error of mean and variance estimates. We demonstrate that, for sparse GP likelihood approximations, we can minimize the pF divergence bounds efficiently. Our experiments show that optimizing the pF divergence bounds has the same computational requirements as variational sparse GPs while providing comparable empirical performance—in addition to our novel finite-data quality guarantees. 

   more » « less
2. Minimizing Convex Functionals over Space of Probability Measures via KL Divergence Gradient Flow

   Yao, Rentian; Huang, Linjun; Yang, Yun (May 2024, Proceedings of Machine Learning Research)

   Motivated by the computation of the non-parametric maximum likelihood estimator (NPMLE) and the Bayesian posterior in statistics, this paper explores the problem of convex optimization over the space of all probability distributions. We introduce an implicit scheme, called the implicit KL proximal descent (IKLPD) algorithm, for discretizing a continuous-time gradient flow relative to the KullbackLeibler divergence for minimizing a convex target functional. We show that IKLPD converges to a global optimum at a polynomial rate from any initialization; moreover, if the objective functional is strongly convex relative to the KL divergence, for example, when the target functional itself is a KL divergence as in the context of Bayesian posterior computation, IKLPD exhibits globally exponential convergence. Computationally, we propose a numerical method based on normalizing flow to realize IKLPD. Conversely, our numerical method can also be viewed as a new approach that sequentially trains a normalizing flow for minimizing a convex functional with a strong theoretical guarantee. 

   more » « less
3. Updating Variational Bayes: fast sequential posterior inference

   https://doi.org/10.1007/s11222-021-10062-2  

   Tomasetti, Nathaniel; Forbes, Catherine; Panagiotelis, Anastasios (February 2022, Statistics and Computing)

   Jasra, Ajay (Ed.)

   Variational Bayesian (VB) methods produce posterior inference in a time frame considerably smaller than traditional Markov Chain Monte Carlo approaches. Although the VB posterior is an approximation, it has been shown to produce good parameter estimates and predicted values when a rich classes of approximating distributions are considered. In this paper, we propose the use of recursive algorithms to update a sequence of VB posterior approximations in an online, time series setting, with the computation of each posterior update requiring only the data observed since the previous update. We show how importance sampling can be incorporated into online variational inference allowing the user to trade accuracy for a substantial increase in computational speed. The proposed methods and their properties are detailed in two separate simulation studies. Additionally, two empirical illustrations are provided, including one where a Dirichlet Process Mixture model with a novel posterior dependence structure is repeatedly updated in the context of predicting the future behaviour of vehicles on a stretch of the US Highway 101. 

   more » « less
4. Pathfinder: Parallel quasi-Newton variational inference

   Zhang, Lu; Carpenter, B; Gelman, A.; Vehtari, A. (January 2022, Journal of machine learning research)

   We propose Pathfinder, a variational method for approximately sampling from differentiable probability densities. Starting from a random initialization, Pathfinder locates normal approximations to the target density along a quasi-Newton optimization path, with local covariance estimated using the inverse Hessian estimates produced by the optimizer. Pathfinder returns draws from the approximation with the lowest estimated Kullback-Leibler (KL) divergence to the target distribution. We evaluate Pathfinder on a wide range of posterior distributions, demonstrating that its approximate draws are better than those from automatic differentiation variational inference (ADVI) and comparable to those produced by short chains of dynamic Hamiltonian Monte Carlo (HMC), as measured by 1-Wasserstein distance. Compared to ADVI and short dynamic HMC runs, Pathfinder requires one to two orders of magnitude fewer log density and gradient evaluations, with greater reductions for more challenging posteriors. Importance resampling over multiple runs of Pathfinder improves the diversity of approximate draws, reducing 1-Wasserstein distance further and providing a measure of robustness to optimization failures on plateaus, saddle points, or in minor modes. The Monte Carlo KL divergence estimates are embarrassingly parallelizable in the core Pathfinder algorithm, as are multiple runs in the resampling version, further increasing Pathfinder's speed advantage with multiple cores. 

   more » « less
5. Asymptotic consistency of loss‐calibrated variational Bayes

   https://doi.org/10.1002/sta4.258  

   Jaiswal, Prateek; Honnappa, Harsha; Rao, Vinayak A. (February 2020, Stat)

   This paper establishes the asymptotic consistency of theloss‐calibrated variational Bayes(LCVB) method. LCVB is a method for approximately computing Bayesian posterior approximations in a “loss aware” manner. This methodology is also highly relevant in general data‐driven decision‐making contexts. Here, we establish the asymptotic consistency of both the loss‐ calibrated approximate posterior and the resulting decision rules. We also establish the asymptotic consistency of decision rules obtained from a “naive” two‐stage procedure that first computes a standard variational Bayes approximation and then uses this in the decision‐making procedure. 

   more » « less