More Like this

1. Adaptive sparse tiling for sparse matrix multiplication

   https://doi.org/10.1145/3293883.3295712  

   Hong, Changwan; Sukumaran-Rajam, Aravind; Nisa, Israt; Singh, Kunal; Sadayappan, P. (January 2019, Proceedings of the 24th ACM Symposium on Principles and Practice of Parallel Programming)
2. Legate Sparse: Distributed Sparse Computing in Python

   https://doi.org/10.1145/3581784.3607033  

   Yadav, Rohan; Lee, Wonchan; Elibol, Melih; Papadakis, Manolis; Lee-Patti, Taylor; Garland, Michael; Aiken, Alex; Kjolstad, Fredrik; Bauer, Michael (November 2023, ACM)
3. Dynamic Sparse Training of Diagonally Sparse Networks.

   Tyagi, Abhishek; Iyer, Arjun; Renninger, William H; Kanan, Christopher; Zhu, Yuhao (May 2025, ICML)
4. Dynamic Sparse Training of Diagonally Sparse Networks

   Tyagi, A; Iyer, A; Renninger, WH; Kanan, C; Zhu, Y (July 2025, Proc. International Conference on Machine Learning (ICML))

   Recent advances in Dynamic Sparse Training (DST) have pushed the frontier of sparse neural network training in structured and unstructured contexts, matching dense-model performance while drastically reducing parameter counts to facilitate model scaling. However, unstructured sparsity often fails to translate into practical speedups on modern hardware. To address this shortcoming, we propose DynaDiag, a novel structured sparse-to-sparse DST method that performs at par with unstructured sparsity. DynaDiag enforces a diagonal sparsity pattern throughout training and preserves sparse computation in forward and backward passes. We further leverage the diagonal structure to accelerate computation via a custom CUDA kernel, rendering the method hardware-friendly. Empirical evaluations on diverse neural architectures demonstrate that our method maintains accuracy on par with unstructured counterparts while benefiting from tangible computational gains. Notably, with 90% sparse linear layers in ViTs, we observe up to a 3.13x speedup in online inference without sacrificing model performance and a 1.59x speedup in training on a GPU compared to equivalent unstructured layers. 

   more » « less
5. Robust Sparse Regularization: Defending Adversarial Attacks Via Regularized Sparse Network

   https://doi.org/10.1145/3386263.3407651  

   Rakin, Adnan Siraj; He, Zhezhi; Yang, Li; Wang, Yanzhi; Wang, Liqiang; Fan, Deliang (September 2020, Proceedings of the 2020 on Great Lakes Symposium on VLSI)

   null (Ed.)

   Deep Neural Network (DNN) trained by the gradient descent method is known to be vulnerable to maliciously perturbed adversarial input, aka. adversarial attack. As one of the countermeasures against adversarial attacks, increasing the model capacity for DNN robustness enhancement was discussed and reported as an effective approach by many recent works. In this work, we show that shrinking the model size through proper weight pruning can even be helpful to improve the DNN robustness under adversarial attack. For obtaining a simultaneously robust and compact DNN model, we propose a multi-objective training method called Robust Sparse Regularization (RSR), through the fusion of various regularization techniques, including channel-wise noise injection, lasso weight penalty, and adversarial training. We conduct extensive experiments to show the effectiveness of RSR against popular white-box (i.e., PGD and FGSM) and black-box attacks. Thanks to RSR, 85 % weight connections of ResNet-18 can be pruned while still achieving 0.68 % and 8.72 % improvement in clean- and perturbed-data accuracy respectively on CIFAR-10 dataset, in comparison to its PGD adversarial training baseline. 

   more » « less