More Like this

1. Supervised Learning via Ensemble Tensor Completion

   https://doi.org/10.1109/IEEECONF51394.2020.9443399  

   Kargas, Nikos; Sidiropoulos, Nicholas D. (November 2020, 2020 Asilomar Conference on on Signals, Systems, and Computers)

   null (Ed.)

   Learning nonlinear functions from input-output data pairs is one of the most fundamental problems in machine learning. Recent work has formulated the problem of learning a general nonlinear multivariate function of discrete inputs, as a tensor completion problem with smooth latent factors. We build upon this idea and utilize two ensemble learning techniques to enhance its prediction accuracy. Ensemble methods can be divided into two main groups, parallel and sequential. Bagging also known as bootstrap aggregation is a parallel ensemble method where multiple base models are trained in parallel on different subsets of the data that have been chosen randomly with replacement from the original training data. The output of these models is usually combined and a single prediction is computed using averaging. One of the most popular bagging techniques is random forests. Boosting is a sequential ensemble method where a sequence of base models are fit sequentially to modified versions of the data. Popular boosting algorithms include AdaBoost and Gradient Boosting. We develop two approaches based on these ensemble learning techniques for learning multivariate functions using the Canonical Polyadic Decomposition. We showcase the effectiveness of the proposed ensemble models on several regression tasks and report significant improvements compared to the single model. 

   more » « less
2. General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

   https://doi.org/10.3389/frai.2021.668353  

   Zhang, Kaiqi; Hawkins, Cole; Zhang, Zheng (January 2022, Frontiers in Artificial Intelligence)

   A major challenge in many machine learning tasks is that the model expressive power depends on model size. Low-rank tensor methods are an efficient tool for handling the curse of dimensionality in many large-scale machine learning models. The major challenges in training a tensor learning model include how to process the high-volume data, how to determine the tensor rank automatically, and how to estimate the uncertainty of the results. While existing tensor learning focuses on a specific task, this paper proposes a generic Bayesian framework that can be employed to solve a broad class of tensor learning problems such as tensor completion, tensor regression, and tensorized neural networks. We develop a low-rank tensor prior for automatic rank determination in nonlinear problems. Our method is implemented with both stochastic gradient Hamiltonian Monte Carlo (SGHMC) and Stein Variational Gradient Descent (SVGD). We compare the automatic rank determination and uncertainty quantification of these two solvers. We demonstrate that our proposed method can determine the tensor rank automatically and can quantify the uncertainty of the obtained results. We validate our framework on tensor completion tasks and tensorized neural network training tasks. 

   more » « less
3. Stable tensor neural networks for efficient deep learning

   https://doi.org/10.3389/fdata.2024.1363978  

   Newman, Elizabeth; Horesh, Lior; Avron, Haim; Kilmer, Misha E (May 2024, Frontiers in Big Data)

   Zhang, Yanqing (Ed.)

   Learning from complex, multidimensional data has become central to computational mathematics, and among the most successful high-dimensional function approximators are deep neural networks (DNNs). Training DNNs is posed as an optimization problem to learn network weights or parameters that well-approximate a mapping from input to target data. Multiway data or tensors arise naturally in myriad ways in deep learning, in particular as input data and as high-dimensional weights and features extracted by the network, with the latter often being a bottleneck in terms of speed and memory. In this work, we leverage tensor representations and processing to efficiently parameterize DNNs when learning from high-dimensional data. We propose tensor neural networks (t-NNs), a natural extension of traditional fully-connected networks, that can be trained efficiently in a reduced, yet more powerful parameter space. Our t-NNs are built upon matrix-mimetic tensor-tensor products, which retain algebraic properties of matrix multiplication while capturing high-dimensional correlations. Mimeticity enables t-NNs to inherit desirable properties of modern DNN architectures. We exemplify this by extending recent work on stable neural networks, which interpret DNNs as discretizations of differential equations, to our multidimensional framework. We provide empirical evidence of the parametric advantages of t-NNs on dimensionality reduction using autoencoders and classification using fully-connected and stable variants on benchmark imaging datasets MNIST and CIFAR-10. 

   more » « less
4. Learning Two-Layer Neural Networks with Symmetric Inputs

   Ge, Rong; Kuditipudi, Rohith; Li, Zhize; Wang, Xiang (July 2019, International Conference on Learning Representations)

   We give a new algorithm for learning a two-layer neural network under a general class of input distributions. Assuming there is a ground-truth two-layer network y = Aσ(Wx) + ξ, where A,W are weight matrices, ξ represents noise, and the number of neurons in the hidden layer is no larger than the input or output, our algorithm is guaranteed to recover the parameters A,W of the ground-truth network. The only requirement on the input x is that it is symmetric, which still allows highly complicated and structured input. Our algorithm is based on the method-of-moments framework and extends several results in tensor decompositions. We use spectral algorithms to avoid the complicated non-convex optimization in learning neural networks. Experiments show that our algorithm can robustly learn the ground-truth neural network with a small number of samples for many symmetric input distributions. 

   more » « less
5. Scalable Computation of $$\mathcal{H}_{\infty}$$ Energy Functions for Polynomial Drift Nonlinear Systems

   https://doi.org/10.23919/ACC60939.2024.10644363  

   Corbin, Nicholas A; Kramer, Boris (July 2024, IEEE)

   This paper presents a scalable tensor-based approach to computing controllability and observability-type energy functions for nonlinear dynamical systems with polynomial drift and linear input and output maps. Using Kronecker product polynomial expansions, we convert the Hamilton- Jacobi-Bellman partial differential equations for the energy functions into a series of algebraic equations for the coefficients of the energy functions. We derive the specific tensor structure that arises from the Kronecker product representation and analyze the computational complexity to efficiently solve these equations. The convergence and scalability of the proposed energy function computation approach is demonstrated on a nonlinear reaction-diffusion model with cubic drift nonlinearity, for which we compute degree 3 energy function approximations in n = 1023 dimensions and degree 4 energy function approximations in n = 127 dimensions. 

   more » « less