 NSFPAR ID:
 10233205
 Date Published:
 Journal Name:
 Part of Advances in Neural Information Processing Systems 33 (NeurIPS 2020)
 Volume:
 33
 Page Range / eLocation ID:
 1750217513
 Format(s):
 Medium: X
 Sponsoring Org:
 National Science Foundation
More Like this

null (Ed.)In this paper, we propose polynomial forms to represent distributions of state variables over time for discretetime stochastic dynamical systems. This problem arises in a variety of applications in areas ranging from biology to robotics. Our approach allows us to rigorously represent the probability distribution of state variables over time, and provide guaranteed bounds on the expectations, moments and probabilities of tail events involving the state variables. First we recall ideas from interval arithmetic, and use them to rigorously represent the state variables at time t as a function of the initial state variables and noise symbols that model the random exogenous inputs encountered before time t. Next we show how concentration of measure inequalities can be employed to prove rigorous bounds on the tail probabilities of these state variables. We demonstrate interesting applications that demonstrate how our approach can be useful in some situations to establish mathematically guaranteed bounds that are of a different nature from those obtained through simulations with pseudorandom numbers.more » « less

We propose new differential privacy solutions for when external invariants and integer constraints are simultaneously enforced on the data product. These requirements arise in real world applications of private data curation, including the public release of the 2020 U.S. Decennial Census. They pose a great challenge to the production of provably private data products with adequate statistical usability. We propose integer subspace differential privacy to rigorously articulate the privacy guarantee when data products maintain both the invariants and integer characteristics, and demonstrate the composition and postprocessing properties of our proposal. To address the challenge of sampling from a potentially highly restricted discrete space, we devise a pair of unbiased additive mechanisms, the generalized Laplace and the generalized Gaussian mechanisms, by solving the Diophantine equations as defined by the constraints. The proposed mechanisms have good accuracy, with errors exhibiting subexponential and subGaussian tail probabilities respectively. To implement our proposal, we design an MCMC algorithm and supply empirical convergence assessment using estimated upper bounds on the total variation distance via Llag coupling. We demonstrate the efficacy of our proposal with applications to a synthetic problem with intersecting invariants, a sensitive contingency table with known margins, and the 2010 Census countylevel demonstration data with mandated fixed state population totals.more » « less

We consider the problem of estimating the mean of a sequence of random elements f (θ, X_1) , . . . , f (θ, X_n) where f is a fixed scalar function, S = (X_1, . . . , X_n) are independent random variables, and θ is a possibly Sdependent parameter. An example of such a problem would be to estimate the generalization error of a neural network trained on n examples where f is a loss function. Classically, this problem is approached through concentration inequalities holding uniformly over compact parameter sets of functions f , for example as in Rademacher or VC type analysis. However, in many problems, such inequalities often yield numerically vacuous estimates. Recently, the PACBayes framework has been proposed as a better alternative for this class of problems for its ability to often give numerically nonvacuous bounds. In this paper, we show that we can do even better: we show how to refine the proof strategy of the PACBayes bounds and achieve even tighter guarantees. Our approach is based on the coinbetting framework that derives the numerically tightest known timeuniform concentration inequalities from the regret guarantees of online gambling algorithms. In particular, we derive the first PACBayes concentration inequality based on the coinbetting approach that holds simultaneously for all sample sizes. We demonstrate its tightness showing that by relaxing it we obtain a number of previous results in a closed form including BernoulliKL and empirical Bernstein inequalities. Finally, we propose an efficient algorithm to numerically calculate confidence sequences from our bound, which often generates nonvacuous confidence bounds even with one sample, unlike the stateoftheart PACBayes bounds.more » « less

The non‐asymptotic tail bounds of random variables play crucial roles in probability, statistics, and machine learning. Despite much success in developing upper bounds on tail probabilities in literature, the lower bounds on tail probabilities are relatively fewer. In this paper, we introduce systematic and user‐friendly schemes for developing non‐asymptotic lower bounds of tail probabilities. In addition, we develop sharp lower tail bounds for the sum of independent sub‐Gaussian and sub‐exponential random variables, which match the classic Hoeffding‐type and Bernstein‐type concentration inequalities, respectively. We also provide non‐asymptotic matching upper and lower tail bounds for a suite of distributions, including gamma, beta, (regular, weighted, and noncentral) chi‐square, binomial, Poisson, Irwin–Hall, etc. We apply the result to establish the matching upper and lower bounds for extreme value expectation of the sum of independent sub‐Gaussian and sub‐exponential random variables. A statistical application of signal identification from sparse heterogeneous mixtures is finally considered.

The development of datainformed predictive models for dynamical systems is of widespread interest in many disciplines. We present a unifying framework for blending mechanistic and machinelearning approaches to identify dynamical systems from noisily and partially observed data. We compare pure datadriven learning with hybrid models which incorporate imperfect domain knowledge, referring to the discrepancy between an assumed truth model and the imperfect mechanistic model as model error. Our formulation is agnostic to the chosen machine learning model, is presented in both continuous and discretetime settings, and is compatible both with model errors that exhibit substantial memory and errors that are memoryless. First, we study memoryless linear (w.r.t. parametricdependence) model error from a learning theory perspective, defining excess risk and generalization error. For ergodic continuoustime systems, we prove that both excess risk and generalization error are bounded above by terms that diminish with the squareroot of T T , the timeinterval over which training data is specified. Secondly, we study scenarios that benefit from modeling with memory, proving universal approximation theorems for two classes of continuoustime recurrent neural networks (RNNs): both can learn memorydependent model error, assuming that it is governed by a finitedimensional hidden variable and that, together, the observed and hidden variables form a continuoustime Markovian system. In addition, we connect one class of RNNs to reservoir computing, thereby relating learning of memorydependent error to recent work on supervised learning between Banach spaces using random features. Numerical results are presented (Lorenz ’63, Lorenz ’96 Multiscale systems) to compare purely datadriven and hybrid approaches, finding hybrid methods less datahungry and more parametrically efficient. We also find that, while a continuoustime framing allows for robustness to irregular sampling and desirable domain interpretability, a discretetime framing can provide similar or better predictive performance, especially when data are undersampled and the vector field defining the true dynamics cannot be identified. Finally, we demonstrate numerically how data assimilation can be leveraged to learn hidden dynamics from noisy, partiallyobserved data, and illustrate challenges in representing memory by this approach, and in the training of such models.more » « less