More Like this

1. Revisiting Pruning at Initialization Through the Lens of Ramanujan Graph

   Hoang, Duc ; Liu, Shiwei ; Marculescu, Radu ; Wang, Zhangyang ( April 2023 , International Conference on Learning Representations)

   Pruning neural networks at initialization (PaI) has received an upsurge of interest due to its end-to-end saving potential. PaI is able to find sparse subnetworks at initialization that can achieve comparable performance to the full networks. These methods can surpass the trivial baseline of random pruning but suffer from a significant performance gap compared to post-training pruning. Previous approaches firmly rely on weights, gradients, and sanity checks as primary signals when conducting PaI analysis. To better understand the underlying mechanism of PaI, we propose to interpret it through the lens of the Ramanujan Graph - a class of expander graphs that are sparse while being highly connected. It is often believed there should be a strong correlation between the Ramanujan graph and PaI since both are about finding sparse and well-connected neural networks. However, the finer-grained link relating highly sparse and connected networks to their relative performance (i.e., ranking of difference sparse structures at the same specific global sparsity) is still missing. We observe that not only the Ramanujan property for sparse networks shows no significant relationship to PaI’s relative performance, but maximizing it can also lead to the formation of pseudo-random graphs with no structural meanings. We reveal the underlying cause to be Ramanujan Graph’s strong assumption on the upper bound of the largest nontrivial eigenvalue (µˆ) of layers belonging to highly sparse networks. We hence propose Iterative Mean Difference of Bound (IMDB) as a mean to relax the µˆ upper bound. Likewise, we also show there exists a lower bound for µˆ, which we call the Normalized Random Coefficient (NaRC), that gives us an accurate assessment for when sparse but highly connected structure degenerates into naive randomness. Finally, we systematically analyze the behavior of various PaI methods and demonstrate the utility of our proposed metrics in characterizing PaI performance. We show that subnetworks preserving better the IMDB property correlate higher in performance, while NaRC provides us with a possible mean to locate the region where highly connected, highly sparse, and non-trivial Ramanujan expanders exist. Our code is available at: https://github.com/VITA-Group/ramanujan-on-pai. 

   more » « less
2. Periodic Signal Denoising: An Analysis-Synthesis Framework Based on Ramanujan Filter Banks and Dictionaries

   https://doi.org/10.1109/ICASSP39728.2021.9413689  

   Kulkarni, Pranav ; Vaidyanathan, P. P. ( June 2021 , Proc. IEEE Int. Conf. Acoust. Speech, and Signal Proc)

   null (Ed.)

   Ramanujan filter banks (RFB) have in the past been used to identify periodicities in data. These are analysis filter banks with no synthesis counterpart for perfect reconstruction of the original signal, so they have not been useful for denoising periodic signals. This paper proposes to use a hybrid analysissynthesis framework for denoising discrete-time periodic signals. The synthesis occurs via a pruned dictionary designed based on the output energies of the RFB analysis filters. A unique property of the framework is that the denoised output signal is guaranteed to be periodic unlike any of the other methods. For a large range of input noise levels, the proposed approach achieves a stable and high SNR gain outperforming many traditional denoising techniques. 

   more » « less
3. DSP-Inspired Deep Learning: A Case Study Using Ramanujan Subspaces

   https://doi.org/10.1109/IEEECONF44664.2019.9048783  

   Tenneti, Srikanth V. ; Vaidyanathan, P. P. ( November 2019 , Proc. Asil. Conf. Sig., Sys., and Comp.)

   Can Deep Learning be used to augment DSP techniques? Algorithms in DSP are typically developed starting from a mathematical model of an application. In some cases however, simplicity of the model can result in deterioration of performance when there is a severe modeling mis-match. This paper explores the idea of implementing a DSP technique as a computational graph, so that hundreds of parameters can jointly be trained to adapt to any given dataset. Using the specific example of period estimation by Ramanujan Subspaces, significant improvement in estimation accuracies under high noise and very short datalengths is demonstrated. 

   more » « less
4. Absence Seizure Detection Using Ramanujan Filter Banks

   https://doi.org/10.1109/ACSSC.2018.8645492  

   Tenneti, Srikanth V. ; Vaidyanathan, P. P. ( October 2018 , Asilomar Conf. on Sig., Sys., and Computers)

   Absence seizures are a type of generalized seizures characterized by a 3 Hz periodic spike and wave discharge pattern in the Electroencephalogram (EEG). The most common way to diagnose them is by detecting such periodic patterns in a patient’s EEG. Recently, a new method known as Ramanujan Filter Bank (RFB) was proposed for identifying, estimating and localizing periodicities in data. The RFB was shown to offer important advantages over traditional period estimation techniques in DSP. In this work, we demonstrate that the RFB offers very useful diagnostic information when applied to EEG signals from absence-seizure patients. 

   more » « less
5. Quadratic ideals and Rogers–Ramanujan recursions

   https://doi.org/10.1007/s11139-018-0127-3  

   Bai, Yuzhe ; Gorsky, Eugene ; Kivinen, Oscar ( January 2019 , The Ramanujan Journal)

   We give an explicit recursive description of the Hilbert series and Gröbner bases for the family of quadratic ideals defining the jet schemes of a double point. We relate these recursions to the Rogers-Ramanujan identity and prove a conjecture of the second author, Oblomkov and Rasmussen. 

   more » « less