More Like this

1. Winning the NIST Contest: A scalable and general approach to differentially private synthetic data

   https://doi.org/10.29012/jpc.778  

   McKenna, Ryan; Miklau, Gerome; Sheldon, Daniel (December 2021, Journal of Privacy and Confidentiality)

   We propose a general approach for differentially private synthetic data generation, that consists of three steps: (1) select a collection of low-dimensional marginals, (2) measure those marginals with a noise addition mechanism, and (3) generate synthetic data that preserves the measured marginals well. Central to this approach is Private-PGM, a post-processing method that is used to estimate a high-dimensional data distribution from noisy measurements of its marginals. We present two mechanisms, NIST-MST and MST, that are instances of this general approach. NIST-MST was the winning mechanism in the 2018 NIST differential privacy synthetic data competition, and MST is a new mechanism that can work in more general settings, while still performing comparably to NIST-MST. We believe our general approach should be of broad interest, and can be adopted in future mechanisms for synthetic data generation. 

   more » « less
2. Moments of Random Quantum Marginals via Weingarten Calculus

   https://doi.org/10.1093/imrn/rnad105  

   Matsumoto, Sho; McSwiggen, Colin (May 2023, International Mathematics Research Notices)

   The randomized quantum marginal problem asks about the joint distribution of the partial traces (“marginals”) of a uniform random Hermitian operator with fixed spectrum acting on a space of tensors. We introduce a new approach to this problem based on studying the mixed moments of the entries of the marginals. For randomized quantum marginal problems that describe systems of distinguishable particles, bosons, or fermions, we prove formulae for these mixed moments, which determine the joint distribution of the marginals completely. Our main tool is Weingarten calculus, which provides a method for computing integrals of polynomial functions with respect to Haar measure on the unitary group. As an application, in the case of two distinguishable particles, we prove some results on the asymptotic behavior of the marginals as the dimension of one or both Hilbert spaces goes to infinity. 

   more » « less
3. Graphical-model based estimation and inference for differential privacy

   McKenna, Ryan; Sheldon, Daniel; Miklau, Gerome (June 2019, Proceedings of Machine Learning Research)

   null (Ed.)

   Many privacy mechanisms reveal high-level information about a data distribution through noisy measurements. It is common to use this information to estimate the answers to new queries. In this work, we provide an approach to solve this estimation problem efficiently using graphical models, which is particularly effective when the distribution is high-dimensional but the measurements are over low-dimensional marginals. We show that our approach is far more efficient than existing estimation techniques from the privacy literature and that it can improve the accuracy and scalability of many state-of-the-art mechanisms. 

   more » « less
4. Generative Marginalization Models

   Liu, Sulin; Ramadge, Peter; Adams, Ryan P (July 2024, MLR)

   We introduce marginalization models (MAMs), a new family of generative models for high-dimensional discrete data. They offer scalable and flexible generative modeling by explicitly modeling all induced marginal distributions. Marginalization models enable fast approximation of arbitrary marginal probabilities with a single forward pass of the neural network, which overcomes a major limitation of arbitrary marginal inference models, such as any-order autoregressive models. MAMs also address the scalability bottleneck encountered in training any-order generative models for high-dimensional problems under the context of energy-based training, where the goal is to match the learned distribution to a given desired probability (specified by an unnormalized log-probability function such as energy or reward function). We propose scalable methods for learning the marginals, grounded in the concept of "marginalization self-consistency". We demonstrate the effectiveness of the proposed model on a variety of discrete data distributions, including images, text, physical systems, and molecules, for maximum likelihood and energy-based training settings. MAMs achieve orders of magnitude speedup in evaluating the marginal probabilities on both settings. For energy-based training tasks, MAMs enable any-order generative modeling of high-dimensional problems beyond the scale of previous methods. 

   more » « less
5. Generative Marginalization Models

   Liu, Sulin; Ramadge, Peter; Adams, Ryan P (July 2023, International Conference on Machine Learning)

   We introduce marginalization models (MaMs), a new family of generative models for high-dimensional discrete data. They offer scalable and flexible generative modeling with tractable likelihoods by explicitly modeling all induced marginal distributions. Marginalization models enable fast evaluation of arbitrary marginal probabilities with a single forward pass of the neural network, which overcomes a major limitation of methods with exact marginal inference, such as autoregressive models (ARMs). We propose scalable methods for learning the marginals, grounded in the concept of "marginalization self-consistency". Unlike previous methods, MaMs support scalable training of any-order generative models for high-dimensional problems under the setting of energy-based training, where the goal is to match the learned distribution to a given desired probability (specified by an unnormalized (log) probability function such as energy function or reward function). We demonstrate the effectiveness of the proposed model on a variety of discrete data distributions, including binary images, language, physical systems, and molecules, for maximum likelihood and energy-based training settings. MaMs achieve orders of magnitude speedup in evaluating the marginal probabilities on both settings. For energy-based training tasks, MaMs enable any-order generative modeling of high-dimensional problems beyond the capability of previous methods. Code is at https://github.com/PrincetonLIPS/MaM. 

   more » « less