More Like this

1. Limiting Spectrum of Randomized Hadamard Transform and Optimal Iterative Sketching Methods

   Lacotte, Jonathan; Liu, Sifan; Dobriban, Edgar; Pilanci, Mert (January 2020, Conference on Neural Information Processing Systems)

   null (Ed.)

   Random projections or sketching are widely used in many algorithmic and learning contexts. Here we study the performance of iterative Hessian sketch for leastsquares problems. By leveraging and extending recent results from random matrix theory on the limiting spectrum of matrices randomly projected with the subsampled randomized Hadamard transform, and truncated Haar matrices, we can study and compare the resulting algorithms to a level of precision that has not been possible before. Our technical contributions include a novel formula for the second moment of the inverse of projected matrices. We also find simple closed-form expressions for asymptotically optimal step-sizes and convergence rates. These show that the convergence rate for Haar and randomized Hadamard matrices are identical, and asymptotically improve upon Gaussian random projections. These techniques may be applied to other algorithms that employ randomized dimension reduction. 

   more » « less
2. High-Dimensional Optimization in Adaptive Random Subspaces

   Lacotte, Jonathan; Pilanci, Mert; Pavone, Marco (December 2019, Advances in neural information processing systems)

   We propose a new randomized optimization method for high-dimensional problems which can be seen as a generalization of coordinate descent to random subspaces. We show that an adaptive sampling strategy for the random subspace significantly outperforms the oblivious sampling method, which is the common choice in the recent literature. The adaptive subspace can be efficiently generated by a correlated random matrix ensemble whose statistics mimic the input data. We prove that the improvement in the relative error of the solution can be tightly characterized in terms of the spectrum of the data matrix, and provide probabilistic upper-bounds. We then illustrate the consequences of our theory with data matrices of different spectral decay. Extensive experimental results show that the proposed approach offers significant speed ups in machine learning problems including logistic regression, kernel classification with random convolution layers and shallow neural networks with rectified linear units. Our analysis is based on convex analysis and Fenchel duality, and establishes connections to sketching and randomized matrix decompositions. 

   more » « less
3. Moments of Random Quantum Marginals via Weingarten Calculus

   https://doi.org/10.1093/imrn/rnad105  

   Matsumoto, Sho; McSwiggen, Colin (May 2023, International Mathematics Research Notices)

   The randomized quantum marginal problem asks about the joint distribution of the partial traces (“marginals”) of a uniform random Hermitian operator with fixed spectrum acting on a space of tensors. We introduce a new approach to this problem based on studying the mixed moments of the entries of the marginals. For randomized quantum marginal problems that describe systems of distinguishable particles, bosons, or fermions, we prove formulae for these mixed moments, which determine the joint distribution of the marginals completely. Our main tool is Weingarten calculus, which provides a method for computing integrals of polynomial functions with respect to Haar measure on the unitary group. As an application, in the case of two distinguishable particles, we prove some results on the asymptotic behavior of the marginals as the dimension of one or both Hilbert spaces goes to infinity. 

   more » « less
4. On the Pseudo-Deterministic Query Complexity of NP Search Problems

   https://doi.org/10.4230/LIPIcs.CCC.2021.36  

   Goldwasser, Shafi and (August 2021, Leibniz international proceedings in informatics)

   Kabanets, Valentine (Ed.)

   We study pseudo-deterministic query complexity - randomized query algorithms that are required to output the same answer with high probability on all inputs. We prove Ω(√n) lower bounds on the pseudo-deterministic complexity of a large family of search problems based on unsatisfiable random CNF instances, and also for the promise problem (FIND1) of finding a 1 in a vector populated with at least half one’s. This gives an exponential separation between randomized query complexity and pseudo-deterministic complexity, which is tight in the quantum setting. As applications we partially solve a related combinatorial coloring problem, and we separate random tree-like Resolution from its pseudo-deterministic version. In contrast to our lower bound, we show, surprisingly, that in the zero-error, average case setting, the three notions (deterministic, randomized, pseudo-deterministic) collapse. 

   more » « less
5. Optimal Randomized First-Order Methods for Least-Squares Problems

   Lacotte, Jonathan; Pilanci, Mert (January 2020, International conference on machine learning)

   null (Ed.)

   We provide an exact analysis of a class of randomized algorithms for solving overdetermined least-squares problems. We consider first-order methods, where the gradients are pre-conditioned by an approximation of the Hessian, based on a subspace embedding of the data matrix. This class of algorithms encompasses several randomized methods among the fastest solvers for leastsquares problems. We focus on two classical embeddings, namely, Gaussian projections and subsampled randomized Hadamard transforms (SRHT). Our key technical innovation is the derivation of the limiting spectral density of SRHT embeddings. Leveraging this novel result, we derive the family of normalized orthogonal polynomials of the SRHT density and we find the optimal pre-conditioned first-order method along with its rate of convergence. Our analysis of Gaussian embeddings proceeds similarly, and leverages classical random matrix theory results. In particular, we show that for a given sketch size, SRHT embeddings exhibits a faster rate of convergence than Gaussian embeddings. Then, we propose a new algorithm by optimizing the computational complexity over the choice of the sketching dimension. To our knowledge, our resulting algorithm yields the best known complexity for solving least-squares problems with no condition number dependence. 

   more » « less