An official website of the United States government
Modern deep neural networks (DNNs) often require high memory consumption and large computational loads. In order to deploy DNN algorithms efficiently on edge or mobile devices, a series of DNN compression algorithms have been explored, including factorization methods. Factorization methods approximate the weight matrix of a DNN layer with the multiplication of two or multiple low-rank matrices. However, it is hard to measure the ranks of DNN layers during the training process. Previous works mainly induce low-rank through implicit approximations or via costly singular value decomposition (SVD) process on every training step. The former approach usually induces a high accuracy loss while the latter has a low efficiency. In this work, we propose SVD training, the first method to explicitly achieve low-rank DNNs during training without applying SVD on every step. SVD training first decomposes each layer into the form of its full-rank SVD, then performs training directly on the decomposed weights. We add orthogonality regularization to the singular vectors, which ensure the valid form of SVD and avoid gradient vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing regularizers on the singular values of each layer. Singular value pruning is applied at the end to explicitly reach a low-rank model. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy, comparing to not only previous factorization methods but also state-of-the-art filter pruning methods.
more »
« less
This content will become publicly available on December 1, 2026
Data-driven low-rank approximation for the electron-hole kernel and acceleration of time-dependent GW calculations
Many-body interactions are essential for understanding non-linear optics and ultrafast spectroscopy of materials. Recent first principles approaches based on nonequilibrium Green’s function formalisms, such as the time-dependent adiabatic GW (TD-aGW) approach, can predict nonequilibrium dynamics of excited states including electron-hole interactions. However, the high-dimensionality of the electron-hole kernel poses significant computational challenges. Here, we develop a data-driven low-rank approximation for the electron-hole kernel, leveraging localized excitonic effects in the Hilbert space of crystalline systems to achieve significant data compression through singular value decomposition (SVD). We show that the subspace of non-zero singular values remains small even as the k-grid grows, ensuring computational tractability with extremely dense k-grids. This low-rank property enables at least 95% data compression and an order-of-magnitude speedup of TD-aGW calculations. Our approach avoids intensive training processes and eliminates time-accumulated errors, seen in previous approaches, providing a general framework for high-throughput, nonequilibrium simulation of light-driven dynamics in materials.
more »
« less
- Award ID(s):
- 2412412
- PAR ID:
- 10617666
- Publisher / Repository:
- NPJ Computational Materials
- Date Published:
- Journal Name:
- npj Computational Materials
- Volume:
- 11
- Issue:
- 1
- ISSN:
- 2057-3960
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
First-principles calculations of electron interactions in materials have seen rapid progress in recent years, with electron-phonon (e-ph) interactions being a prime example. However, these techniques use large matrices encoding the interactions on dense momentum grids, which reduces computational efficiency and obscures interpretability. For e-ph interactions, existing interpolation techniques leverage locality in real space, but the high dimensionality of the data remains a bottleneck to balance cost and accuracy. Here we show an efficient way to compress e-ph interactions based on singular value decomposition (SVD), a widely used matrix and image compression technique. Leveraging (un)constrained SVD methods, we accurately predict material properties related to e-ph interactions—including charge mobility, spin relaxation times, band renormalization, and superconducting critical temperature—while using only a small fraction (1%–2%) of the interaction data. These findings unveil the hidden low-dimensional nature of e-ph interactions. Furthermore, they accelerate state-of-the-art first-principles e-ph calculations by about 2 orders of magnitude without sacrificing accuracy. Our Pareto-optimal parametrization of e-ph interactions can be readily generalized to electron-electron and electron-defect interactions, as well as to other couplings, advancing quantitative studies of condensed matter.more » « less
-
In this paper, we present a generalized, data-driven collisional operator for one-component plasmas, learned from molecular dynamics simulations, to extend the collisional kinetic model beyond the weakly coupled regime. The proposed operator features an anisotropic, non-stationary collision kernel that accounts for particle correlations typically neglected in classical Landau formulations. To enable efficient numerical evaluation, we develop a fast spectral separation method that represents the kernel as a low-rank tensor product of univariate basis functions. This formulation admits an O(N log N) algorithm via fast Fourier transforms and preserves key physical properties, including discrete conservation laws and the H-theorem, through a structure-preserving central difference discretization. Numerical experiments demonstrate that the proposed model accurately captures plasma dynamics in the moderately coupled regime beyond the standard Landau model while maintaining high computational efficiency and structure-preserving properties.more » « less
-
We propose a data-driven approach for deep convolutional neural network compression that achieves high accuracy with high throughput and low memory requirements. Current network compression methods either find a low-rank factorization of the features that requires more memory, or select only a subset of features by pruning entire filter channels. We propose the Cascaded Projection (CaP) compression method that projects the output and input filter channels of successive layers to a unified low dimensional space based on a low-rank projection. We optimize the projection to minimize classification loss and the difference between the next layer’s features in the compressed and uncompressed networks. To solve this non-convex optimization problem we propose a new optimization method of a proxy matrix using back propagation and Stochastic Gradient Descent (SGD) with geometric constraints. Our cascaded projection approach leads to improvements in all critical areas of network compression: high accuracy, low memory consumption, low parameter count and high processing speed. The proposed CaP method demonstrates state-of-the-art results compressing VGG16 and ResNet networks with over 4× reduction in the number of computations and excellent performance in top-5 accuracy on the ImageNet dataset before and after fine-tuning.more » « less
-
Two-dimensional Raman and hybrid terahertz-Raman spectroscopic techniques provide invaluable insight into molecular structures and dynamics of condensed-phase systems. However, corroborating experimental results with theory is difficult due to the high computational cost of incorporating quantum-mechanical effects in the simulations. Here, we present the equilibrium–nonequilibrium ring-polymer molecular dynamics (RPMD), a practical computational method that can account for nuclear quantum effects on the two-time response function of nonlinear optical spectroscopy. Unlike a recently developed approach based on the double Kubo transformed (DKT) correlation function, our method is exact in the classical limit, where it reduces to the established equilibrium-nonequilibrium classical molecular dynamics method. Using benchmark model calculations, we demonstrate the advantages of the equilibrium–nonequilibrium RPMD over classical and DKT-based approaches. Importantly, its derivation, which is based on the nonequilibrium RPMD, obviates the need for identifying an appropriate Kubo transformed correlation function and paves the way for applying real-time path-integral techniques to multidimensional spectroscopy.more » « less
