More Like this

1. Nonlinear Dependent Component Analysis: Identifiability and Algorithm

   https://doi.org/10.23919/Eusipco47968.2020.9287354  

   Lyu, Qi; Fu, Xiao (January 2021, EUSIPCO 2020)

   null (Ed.)

   This work studies the model identification problem of a class of post-nonlinear mixture models in the presence of dependent latent components. Particularly, our interest lies in latent components that are nonnegative and sum-to-one. This problem is motivated by applications such as hyperspectral unmixing under nonlinear distortion effects. Many prior works tackled nonlinear mixture analysis using statistical independence among the latent components, which is not applicable in our case. A recent work by Yang et al. put forth a solution for this problem leveraging functional equations. However, the identifiability conditions derived there are somewhat restrictive. The associated implementation also has difficulties-the function approximator used in their work may not be able to represent general nonlinear distortions and the formulated constrained neural network optimization problem may be challenging to handle. In this work, we advance both the theoretical and practical aspects of the problem of interest. On the theory side, we offer a new identifiability condition that circumvents a series of stringent assumptions in Yang et al.'s work. On the algorithm side, we propose an easy-to-implement unconstrained neural network-based algorithm-without sacrificing function approximation capabilities. Numerical experiments are employed to support our design. 

   more » « less
2. Heavy-tailed Streaming Statistical Estimation

   Tsai, Che-Ping; Prasad, Adarsh; Balakrishnan, Sivaraman; Ravikumar, Pradeep (March 2022, International Conference on Artificial Intelligence and Statistics (AISTATS))

   We consider the task of heavy-tailed statistical estimation given streaming p-dimensional samples. This could also be viewed as stochastic optimization under heavy-tailed distributions, with an additional O(p) space complexity constraint. We design a clipped stochastic gradient descent algorithm and provide an improved analysis, under a more nuanced condition on the noise of the stochastic gradients, which we show is critical when analyzing stochastic optimization problems arising from general statistical estimation problems. Our results guarantee convergence not just in expectation but with exponential concentration, and moreover does so using O(1) batch size. We provide consequences of our results for mean estimation and linear regression. Finally, we provide empirical corroboration of our results and algorithms via synthetic experiments for mean estimation and linear regression. 

   more » « less
3. Consistent Estimation of Identifiable Nonparametric MixtureModels from Grouped Observations

   Ritchie, Alexander; Vandermeulen, Robert; Scott, Clayton (January 2020, Advances in neural information processing systems)

   null (Ed.)

   Recent research has established sufficient conditions for finite mixture models to be identifiablefrom grouped observations. These conditions allow the mixture components to be nonparametricand have substantial (or even total) overlap. This work proposes an algorithm that consistentlyestimates any identifiable mixture model from grouped observations. Our analysis leverages anoracle inequality for weighted kernel density estimators of the distribution on groups, togetherwith a general result showing that consistent estimation of the distribution on groups impliesconsistent estimation of mixture components. A practical implementation is provided for pairedobservations, and the approach is shown to outperform existing methods, especially when mixturecomponents overlap significantly. 

   more » « less
4. Consistent Estimation of Identifiable Nonparametric Mixture Models from Grouped Observations

   Ritchie, Alexander; Vandermeulen, Robert A.; Scott, Clayton (January 2020, 34th Conferene on Neural Information Processing Systems (NeurIPS 2020))

   null (Ed.)

   Recent research has established sufficient conditions for finite mixture models to be identifiable from grouped observations. These conditions allow the mixture components to be nonparametric and have substantial (or even total) overlap. This work proposes an algorithm that consistently estimates any identifiable mixture model from grouped observations. Our analysis leverages an oracle inequality for weighted kernel density estimators of the distribution on groups, together with a general result showing that consistent estimation of the distribution on groups implies consistent estimation of mixture components. A practical implementation is provided for paired observations, and the approach is shown to outperform existing methods, especially when mixture components overlap significantly. 

   more » « less
5. Learning Mixtures of Smooth Product Distributions: Identifiability and Algorithm

   Nikos Kargas, Nicholas D. (April 2019, Proceedings AISTATS)

   We study the problem of learning a mixture model of non-parametric product distributions. The problem of learning a mixture model is that of finding the component distributions along with the mixing weights using observed samples generated from the mixture. The problem is well-studied in the parametric setting, i.e., when the component distributions are members of a parametric family - such as Gaussian distributions. In this work, we focus on multivariate mixtures of non-parametric product distributions and propose a two-stage approach which recovers the component distributions of the mixture under a smoothness condition. Our approach builds upon the identifiability properties of the canonical polyadic (low-rank) decomposition of tensors, in tandem with Fourier and Shannon-Nyquist sampling staples from signal processing. We demonstrate the effectiveness of the approach on synthetic and real datasets. 

   more » « less