More Like this

1. Consistent Estimation of Identifiable Nonparametric MixtureModels from Grouped Observations

   Ritchie, Alexander; Vandermeulen, Robert; Scott, Clayton (January 2020, Advances in neural information processing systems)

   null (Ed.)

   Recent research has established sufficient conditions for finite mixture models to be identifiablefrom grouped observations. These conditions allow the mixture components to be nonparametricand have substantial (or even total) overlap. This work proposes an algorithm that consistentlyestimates any identifiable mixture model from grouped observations. Our analysis leverages anoracle inequality for weighted kernel density estimators of the distribution on groups, togetherwith a general result showing that consistent estimation of the distribution on groups impliesconsistent estimation of mixture components. A practical implementation is provided for pairedobservations, and the approach is shown to outperform existing methods, especially when mixturecomponents overlap significantly. 

   more » « less
2. Adaptive Bayesian Estimation of Discrete‐Continuous Distributions Under Smoothness and Sparsity

   https://doi.org/10.3982/ECTA17884  

   Norets, Andriy; Pelenis, Justinas (January 2022, Econometrica)

   We consider nonparametric estimation of a mixed discrete‐continuous distribution under anisotropic smoothness conditions and a possibly increasing number of support points for the discrete part of the distribution. For these settings, we derive lower bounds on the estimation rates. Next, we consider a nonparametric mixture of normals model that uses continuous latent variables for the discrete part of the observations. We show that the posterior in this model contracts at rates that are equal to the derived lower bounds up to a log factor. Thus, Bayesian mixture of normals models can be used for (up to a log factor) optimal adaptive estimation of mixed discrete‐continuous distributions. The proposed model demonstrates excellent performance in simulations mimicking the first stage in the estimation of structural discrete choice models. 

   more » « less
3. Identifiability of Nonparametric Mixture Models and Bayes Optimal Clustering

   https://doi.org/10.1214/19-AOS1887  

   Aragam, B.; Dan, C.; Xing, E.; Ravikumar, P. (April 2020, The annals of statistics)

   Motivated by problems in data clustering, we establish general conditions under which families of nonparametric mixture models are identifiable, by introducing a novel framework involving clustering overfitted parametric (i.e. misspecified) mixture models. These identifiability conditions generalize existing conditions in the literature, and are flexible enough to include for example mixtures of Gaussian mixtures. In contrast to the recent literature on estimating nonparametric mixtures, we allow for general nonparametric mixture components, and instead impose regularity assumptions on the underlying mixing measure. As our primary application, we apply these results to partition-based clustering, generalizing the notion of a Bayes optimal partition from classical parametric model-based clustering to nonparametric settings. Furthermore, this framework is constructive so that it yields a practical algorithm for learning identified mixtures, which is illustrated through several examples on real data. The key conceptual device in the analysis is the convex, metric geometry of probability measures on metric spaces and its connection to the Wasserstein convergence of mixing measures. The result is a flexible framework for nonparametric clustering with formal consistency guarantees. 

   more » « less
4. Partial disentanglement for domain adaptation

   Kong, Lingjing; Xie, Shaoan; Yao, Weiran; Zheng, Yujia; Chen, Guangyi; Stojanov, Petar; Akinwande, Victor; Zhang, Kun (July 2022, Proceedings of Machine Learning Research)

   Chaudhuri, Kamalika; Jegelka, Stefanie; Song, Le; Szepesvari, Csaba; Niu, Gang; Sabato, Sivan (Ed.)

   Unsupervised domain adaptation is critical to many real-world applications where label information is unavailable in the target domain. In general, without further assumptions, the joint distribution of the features and the label is not identifiable in the target domain. To address this issue, we rely on a property of minimal changes of causal mechanisms across domains to minimize unnecessary influences of domain shift. To encode this property, we first formulate the data generating process using a latent variable model with two partitioned latent subspaces: invariant components whose distributions stay the same across domains, and sparse changing components that vary across domains. We further constrain the domain shift to have a restrictive influence on the changing components. Under mild conditions, we show that the latent variables are partially identifiable, from which it follows that the joint distribution of data and labels in the target domain is also identifiable. Given the theoretical insights, we propose a practical domain adaptation framework, called iMSDA. Extensive experimental results reveal that iMSDA outperforms state-of-the-art domain adaptation algorithms on benchmark datasets, demonstrating the effectiveness of our framework. 

   more » « less
5. Mixture Proportion Estimation Beyond Irreducibility

   Zhu, Yillun; Fjeldsted, Aaron; Holland, Darren; Landon, George; Lintereur, Azaree; Scott, Clayton (February 2023, Proceedings of Machine Learning Research)

   The task of mixture proportion estimation (MPE) is to estimate the weight of a component distribution in a mixture, given observations from both the component and mixture. Previous work on MPE adopts the irreducibility assumption, which ensures identifiablity of the mixture proportion. In this paper, we propose a more general sufficient condition that accommodates several settings of interest where irreducibility does not hold. We further present a resampling-based meta algorithm that takes any existing MPE algorithm designed to work under irreducibility and adapts it to work under our more general condition. Our approach empirically exhibits improved estimation performance relative to baseline methods and to a recently proposed regrouping-based algorithm. 

   more » « less