More Like this

1. Adaptive sparse tiling for sparse matrix multiplication

   https://doi.org/10.1145/3293883.3295712  

   Hong, Changwan; Sukumaran-Rajam, Aravind; Nisa, Israt; Singh, Kunal; Sadayappan, P. (January 2019, Proceedings of the 24th ACM Symposium on Principles and Practice of Parallel Programming)
2. Legate Sparse: Distributed Sparse Computing in Python

   https://doi.org/10.1145/3581784.3607033  

   Yadav, Rohan; Lee, Wonchan; Elibol, Melih; Papadakis, Manolis; Lee-Patti, Taylor; Garland, Michael; Aiken, Alex; Kjolstad, Fredrik; Bauer, Michael (November 2023, ACM)
3. Dynamic Sparse Training of Diagonally Sparse Networks.

   Tyagi, Abhishek; Iyer, Arjun; Renninger, William H; Kanan, Christopher; Zhu, Yuhao (May 2025, ICML)
4. Dynamic Sparse Training of Diagonally Sparse Networks

   Tyagi, A; Iyer, A; Renninger, WH; Kanan, C; Zhu, Y (July 2025, Proc. International Conference on Machine Learning (ICML))

   Recent advances in Dynamic Sparse Training (DST) have pushed the frontier of sparse neural network training in structured and unstructured contexts, matching dense-model performance while drastically reducing parameter counts to facilitate model scaling. However, unstructured sparsity often fails to translate into practical speedups on modern hardware. To address this shortcoming, we propose DynaDiag, a novel structured sparse-to-sparse DST method that performs at par with unstructured sparsity. DynaDiag enforces a diagonal sparsity pattern throughout training and preserves sparse computation in forward and backward passes. We further leverage the diagonal structure to accelerate computation via a custom CUDA kernel, rendering the method hardware-friendly. Empirical evaluations on diverse neural architectures demonstrate that our method maintains accuracy on par with unstructured counterparts while benefiting from tangible computational gains. Notably, with 90% sparse linear layers in ViTs, we observe up to a 3.13x speedup in online inference without sacrificing model performance and a 1.59x speedup in training on a GPU compared to equivalent unstructured layers. 

   more » « less
5. Learning mixtures of sparse linear regressions using sparse graph codes

   https://doi.org/10.1109/ALLERTON.2017.8262790  

   Yin, Dong; Pedarsani, Ramtin; Chen, Yudong; Ramchandran, Kannan (October 2017, 55th Annual Allerton Conference on Communication, Control, and Computing (Allerton))