<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training</title></titleStmt>
			<publicationStmt>
				<publisher>Neural Information Processing Systems Foundation, Inc.</publisher>
				<date>06/05/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10614842</idno>
					<idno type="doi"></idno>
					<title level='j'>Advances in neural information processing systems</title>
<idno>1049-5258</idno>
<biblScope unit="volume"></biblScope>
<biblScope unit="issue"></biblScope>					

					<author>Jie Ji</author><author>Gen Li</author><author>Fatemeh Afghah</author><author>Linke Guo</author><author>Xiaoyong Yuan</author><author>Xiaolong Ma</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Sparse training stands as a landmark approach in addressing the considerable training resource demands imposed by the continuously expanding size of Deep Neural Networks (DNNs). However, the training of a sparse DNN encounters great challenges in achieving optimal generalization ability despite the efforts from the state-of-the-art sparse training methodologies. To unravel the mysterious reason behind the difficulty of sparse training, we connect network sparsity with the structure of neural loss functions and identify that the cause of such difficulty lies in a chaotic loss surface. In light of such revelation, we propose S 2 -SAM, characterized by a Single-step Sharpness-Aware Minimization that is tailored for Sparse training. For the first time, S 2 -SAM innovates the traditional SAM-style optimization by approximating sharpness perturbation through prior gradient information, incurring zero extra cost. Therefore, S 2 -SAM not only exhibits the capacity to improve generalization but also aligns with the efficiency goal of sparse training. Additionally, we study the generalization result of S 2 -SAM and provide theoretical proof for convergence. Through extensive experiments, S 2 -SAM demonstrates its universally applicable plug-and-play functionality, enhancing accuracy across various sparse training methods. Code available at https://github.com/jjsrf/SSAM-NEURIPS2024.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>The arrival of the Artificial General Intelligence (AGI) <ref type="bibr">[1]</ref> era has urged an ever-expanding realm of artificial intelligence, bringing significant growth in deep neural networks (DNNs) depth and intricacy. The efficient training of DNN has thus emerged as an uttermost imperative, demanding immediate and concerted efforts.</p><p>To train an overparameterized, large-scale DNN efficiently while preserving its high accuracy, stateof-the-art literature has introduced sparse training <ref type="bibr">[2]</ref><ref type="bibr">[3]</ref><ref type="bibr">[4]</ref><ref type="bibr">[5]</ref><ref type="bibr">[6]</ref> as a straightforward solution that reduces both parameter footprint and computation cost. With the presence of a large portion of zeros in the network, both forward and backward computation for training can be saved by skipping those zeros, as well as reducing the memory consumption since the zeros are not necessary to be stored. However, training a sparse neural network is difficult since the optimization under sparse regime readily converges to stationary points with a sub-optimal generalization accuracy (i.e., saddle points) <ref type="bibr">[7]</ref>. Due to the difficulty of directly (i.e., without the time-consuming linear interpolation) assessing if the sparse solution is in the saddle point, finding workaround approaches for training a better sparse neural network is critical, especially for the practical usage in efficient learning <ref type="bibr">[8]</ref><ref type="bibr">[9]</ref><ref type="bibr">[10]</ref><ref type="bibr">[11]</ref><ref type="bibr">[12]</ref><ref type="bibr">[13]</ref><ref type="bibr">[14]</ref><ref type="bibr">[15]</ref><ref type="bibr">[16]</ref><ref type="bibr">[17]</ref>.</p><p>To address the above difficulty, many efforts have been made. For instance, the Static Sparse Training (SST) such as the Lottery Ticket Hypothesis (LTH) <ref type="bibr">[20]</ref>, SNIP <ref type="bibr">[21]</ref>, GraSP <ref type="bibr">[18]</ref> and SynFlow <ref type="bibr">[22]</ref>  The loss surface visualization for training a sparse neural network using ResNet-32 on CIFAR-10. We select two representative sparse training methods <ref type="bibr">[18,</ref><ref type="bibr">3]</ref> and incorporate different levels of sparsity. We also quantify the loss surface behavior using coefficient Ra <ref type="bibr">[19]</ref> to evaluate sharpness. With increased sparsity, Ra becomes larger, indicating sharper and steeper surface. determine a static sparse pattern at training initialization. On the other hand, Dynamic Sparse Training (DST) such as SET <ref type="bibr">[2]</ref>, RigL <ref type="bibr">[23]</ref>, and MEST <ref type="bibr">[3]</ref> iteratively updates the sparse topology during training to find a better sparse model. However, those methods either suffer from suboptimal generalization ability due to heuristic nature of their methodology or metric settings, or experience difficulties in parameter setting and dynamic sparsity scheduling. With the increase of the sparsity, these phenomenon become more severe.</p><p>We identify that the cause of the sparse network learning difficulties lies in the variable learning dynamics, which is closely related to the network topology. When incorporating sparsity into a neural network, the effective structure of the original network becomes narrower. According to the study of neural loss function structure <ref type="bibr">[24]</ref>, a wide network resulted in flat minima and wide regions of apparent convexity, which helps prevent the chaotic behavior that occurs during training. Evidently, higher sparsity indicates a narrower structure, which suggests more chaotic behavior is expected during training, thus degrading the accuracy. We perform different types of sparse training with different levels of sparsity and plot the loss surface in Figure <ref type="figure">1</ref> to demonstrate the convexity of the training behavior. According to the sharpness of the loss basin, we observe a transition from a smooth and close-to-convex surface to a steep one as sparsity is introduced and increased. Since higher sparsity levels lead to more chaotic behavior, in landscapes that are narrow and sharp with a large Ra value, we might expect to encounter more chaotic training behavior that degrades generalization ability.</p><p>It seems that achieving coexistence of sparsity and good generalization ability during training is challenging. Therefore, we raise the following question: is there a simple, effective method that can improve generalization ability of training a sparse neural network, without sacrificing efficiency (sparsity) and incurring zero extra cost? We believe the answer to the above question lies in the sharpness of the loss surface. Inspired by the Sharpness-Aware Minimization (SAM) <ref type="bibr">[25]</ref> technique that finds flatter minima that have uniformly low loss in nearby regions, we argue that such technique is especially suitable for sparse training since the steep loss surface induced by sparsity can be directly mitigated. However, to leverage sharpness for better generalization, SAM uses an additional full training step (i.e., forward and backward propagation) to quantify and evaluate loss on the summation of current weights and a constrained perturbation, which roughly doubles the computation of training. Such extra costs contradict the goal of sparse training. Although some literature <ref type="bibr">[26,</ref><ref type="bibr">27]</ref> have been proposed to reduce the computation cost for SAM, the computation is still significant, posing a roadblock to the wide application of SAM to the realm of sparse training.</p><p>In this paper, we propose a novel approach to achieve sharpness-aware minimization with zero extra cost, tailored for the sparse training regime to maintain efficiency while improving generalization ability. It is the first time that a Single-step Sharpness-Aware Minimization is proposed for Sparse training (S 2 -SAM). Different from the traditional two-step computation regime of SAM, S 2 -SAM uses a unique single-step approach that leverages sharpness and trains weights with one training step. Specifically, S 2 -SAM uses the weight gradients from the prior step to approximate the perturbation to the weights, thus solving the sharpness evaluation without performing an extra full training step. Therefore, S 2 -SAM incurs zero extra cost to achieve sharpness-aware training, which aligns with the efficiency goal of sparse training. We also study the generalization result of S 2 -SAM and provide theoretical proof for convergence. We demonstrate that S 2 -SAM provides a straightforward plug-andplay functionality on variety of sparse training methods, significantly boosting their accuracy. Our contributions are summarized as follows:</p><p>&#8226; We identify that the difficulty of training a sparse neural network lies in the increasingly chaotic and steep loss surface when sparsity is introduced and increased.</p><p>&#8226; We develop a novel Single-step Sharpness-Aware Minimization technique tailored for Sparse training (S 2 -SAM), and it is the first time that a SAM-style optimization with zero extra computation cost has been proposed. &#8226; We study the generalization result of S 2 -SAM and provide theoretical proof to demonstrate that the S 2 -SAM is guaranteed for convergence. &#8226; Through systematic evaluations, we show that S 2 -SAM provides a plug-and-play functionality applied to a variety of sparse training methods, and consistently improves accuracy on different networks and datasets. 2 Proposed Method 2.1 Preliminary of Sharpness-Aware Minimization w t-1 w t w t w ) (S -SAM) 2 (Original update) -&#951; (g + &#955; t t t-1 &#961;g t-1 g t-1 w t-2 u t SAM is an optimization technique designed to enhance neural network generalization and mitigate overfitting. It minimizes the maximum loss in a neighborhood around the current parameters, as opposed to solely focusing on the loss at the current point. This approach identifies flatter minima that have uniformly low loss in nearby regions, ultimately contributing to improved generalization performance.</p><p>Specifically, consider a family of models parameterized by w &#8712; W &#8838; R d ; L is the loss function, and S denotes the training dataset. SAM aims to minimize the following upper bound of the PAC-Bayesian generalization error: for any &#961; &gt; 0,</p><p>To solve the above minimax problem, at each iteration t, SAM updates the following steps:</p><p>where 1/p + 1/q = 1, &#961; &gt; 0 is a hyperparameter, &#955; &gt; 0 is the parameter for weight decay, and &#951; t &gt; 0 is the learning rate. By setting p = q = 2 and introducing an intermediate variable u t , we have:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.2">The Proposed S 2 -SAM Method</head><p>As shown in Equation <ref type="bibr">(2)</ref>, SAM needs to compute the gradient twice at each iteration, involving additional computation costs. Further, the two-step gradient computation is not parallelizable, which presents challenges for deployment in large-scale training scenarios. Thus, we propose a new algorithm that only needs to compute the gradient once in each iteration. Different from prior efficient SAM works <ref type="bibr">[26]</ref><ref type="bibr">[27]</ref><ref type="bibr">[28]</ref> aiming to reduce the computation by introducing periodically SAM steps or data selection, our proposed framework S 2 -SAM achieves zero extra computation cost while maintaining the improved generalization ability of sparse network training.</p><p>In Figure <ref type="figure">2</ref>, we demonstrate that the perturbation on the current weights is approximated by the weight gradients g t-1 from the prior step. The rational behind such design is that the gradient direction (i.e., w t-2 to w t-1 ) from prior step is optimizing the prior loss to a considerable extent, thus it can be used to represent a sharp direction among all perturbation directions at t -1. Since the loss surface of a sparse network is too chaotic, a perturbation with relatively high degree of sharpness (i.e., the prior gradient) can still find a neighborhood with near-maximum loss. The following equations specifying g t-1 information are used to substitute the &#8711;L S (w t-1 ) step in Equation <ref type="bibr">(2)</ref>. Thus, we only need to compute the gradient once in each iteration:</p><p>Remark 1. The parameter &#961; can vary in terms of iteration t : &#961; t = c/t, where c &gt; 0 is a constant.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.3">Generalization Analysis</head><p>In this section, we study the generalization result of S 2 -SAM. First, we give some notations, where most of them are followed by <ref type="bibr">[29,</ref><ref type="bibr">30]</ref>. Let w S = A(S) be a solution that generated by a random algorithm A based on dataset S. Recall that problem (8)</p><p>is called empirical risk minimization in the literature, and the true risk minimization is given by min w&#8712;W</p><p>We define its optimal solution: w * &#8712; arg min w&#8712;W F (w). Then the excess risk bound (ERB) is defined as E A,S [F (w S )] -F (w * ) . (10) It has been shown that the ERB can be upper bounded by optimization error and generalization error <ref type="bibr">[29,</ref><ref type="bibr">30]</ref>. We notice that there are several works <ref type="bibr">[29,</ref><ref type="bibr">30]</ref> studying the generalization result of SGD for non-convex setting under different conditions, such as bounded stochastic gradient &#8741;&#8711; w &#8467;(y, f (w; x))&#8741; &#8804; G and decaying learning rate &#951; t &#8804; c t with a constant c &gt; 0, where t is the optimization iteration. In this paper, we are not interested in establishing a fast rate in ERB under different conditions, but we want to explore the generalization ability of S 2 -SAM with the fewest possible modifications when building a bridge between theory and practice. For example, weight decay is a widely used trick when training deep neural networks. With the use of weight decay, the empirical risk minimization in practice becomes min w&#8712;W F S (w) := F S (w) + &#955; 2 &#8741;w&#8741; 2 . Then, we define some notations as follows. Specifically, let</p><p>Followed by <ref type="bibr">[30]</ref>, we use the following decomposition of testing error:</p><p>) where the upper bound is the optimization error plus the generalization error.</p><p>Next, we present some notations and assumptions that will be used in the convergence analysis. Throughout this paper, we also make the following assumptions for solving the problem <ref type="bibr">(8)</ref>.</p><p>Assumption 1. Assume the following conditions hold: (i) The stochastic gradient of F S (w) is unbiased, i.e., E (x,y) [&#8711;&#8467;(y, f (w; x))] = &#8711;F S (w), and the variance of stochastic gradient is bounded, i.e., there exists a constant &#963; 2 &gt; 0, such that</p><p>(ii) F S (w) is smooth with an L-Lipchitz continuous gradient, i.e., it is differentiable and there exists a constant L &gt; 0 such that &#8741;&#8711;F S (w) -&#8711;F S (u)&#8741; &#8804; L&#8741;w -u&#8741;, &#8704;w, u &#8712; W.</p><p>Assumption 2. There exists a constant &#181; &gt; 0 such that 2&#181; (F S (w) -F S (w * S )) &#8804; &#8741;&#8711;F S (w)&#8741; 2 , &#8704;w &#8712; W, where w * S &#8712; min w&#8712;W F S (w) is a optimal solution (PL condition <ref type="bibr">[31]</ref>). Now consider that A = S 2 -SAM. We define the gradient update rule G &#8467;,&#951; as follows</p><p>Then we have the following lemma, which is similar to Lemma 2.5 and Lemma 4.2 in <ref type="bibr">[29]</ref> that use recursive definition through variable G &#8242; and w</p><p>Please see the proof of Lemma 1 in Appendix B.</p><p>Theorem 1. Under Assumption 1, assume that &#8467;(y, f (w, x)) is L-smooth and B-Lipschitz, suppose F S (w) satisfies Assumption 2 and min w&#8712;W F S (w) &#8804; F S (w * S ) + &#955; 2 &#8741;w t &#8741; 2 with &#955; = 2L, where w t is the intermediate solution of A, then</p><p>where A is SGD.</p><p>The w * S &#8712; min w&#8712;W F S (w) is an optimal solution, and we show that the generalization error is bounded. Based on Lemma 1, the proof of Theorem 1 is derived in Appendix C.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Experimental Results</head><p>In this section, we carry out comprehensive experiments to demonstrate how S 2 -SAM improves sparse training performance. We test S 2 -SAM on CIFAR-10/100 <ref type="bibr">[34]</ref> with ResNet-32 <ref type="bibr">[32]</ref> and VGG-19 <ref type="bibr">[35]</ref>, and we also perform experiments on ImageNet-1K <ref type="bibr">[36]</ref> and ImageNet-C <ref type="bibr">[37]</ref> based on ResNet-50 <ref type="bibr">[38]</ref>.</p><p>Following the recent developments in sparse training techniques, we apply S 2 -SAM on static sparse training such as LTH <ref type="bibr">[20]</ref>, SNIP <ref type="bibr">[21]</ref>, GraSP <ref type="bibr">[18]</ref>, as well as dynamic sparse training methods such as SET <ref type="bibr">[2]</ref>, DSR <ref type="bibr">[33]</ref>, RigL <ref type="bibr">[23]</ref>, MEST <ref type="bibr">[3]</ref>, CHEX <ref type="bibr">[10]</ref> and Chase <ref type="bibr">[39]</ref>. We apply S 2 -SAM to the official codes or published implementations to show performance gains. All of our experiments are performed on NVIDIA 4&#215; A6000 GPUs. We repeat training experiments for 3 times and report the mean and standard deviation of the accuracy. For training throughput evaluation, we adopt the original settings of each baseline method and record the throughput on 4&#215; A6000 GPUs. We stress that for all experiments, we only use one setting for S 2 -SAM, which indicates an easy implementation that has the potential for wide applicability. We also find that S 2 -SAM works better on high sparsity. For all sparse training with a 98% sparsity ratio, S 2 -SAM improves the original baseline accuracy by an average of 0.77% on CIFAR-10 and 1.12% on CIFAR-100, which shows S 2 -SAM is especially suitable for challenging tasks.</p><p>ImageNet-1K. We evaluate S 2 -SAM on ImageNet-1K dataset. The results are shown in Table <ref type="table">2</ref>. We report the top-1 accuracy of the sparse training results, as well the overall training cost with FLOPs. We set hyper-parameter &#961; = 1.0 for all experiments. Compared to the baseline methods, our result shows higher test accuracy at the same computational cost. Similarly, S 2 -SAM also shows better gain on a higher sparsity ratio for most of the sparse training baselines, which indicates the consistency of using our methods on different scales of datasets. To make a fair comparison with different training recipes, we slightly scale up our training epochs to have the same or less overall training FLOPs to compare with longer training baselines, and we find out that S 2 -SAM shows stable improvement on those methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.2">Sparse Training with Structured Sparsity</head><p>In this section, we evaluate how S 2 -SAM performs on structured sparse training methods. Note that all previous baselines use unstructured sparsity where the zeros are scattered in the network. Such unstructured sparse training shows the potential of training a sparse model, but cannot fully represent training acceleration ability. Therefore, we apply S 2 -SAM to two representative structured sparse trainings, CHEX <ref type="bibr">[10]</ref> and Chase <ref type="bibr">[39]</ref> which dynamically train sparse networks with channel-level sparsity. The results are shown in Table <ref type="table">3</ref>. From the results, we can see that the average accuracy improvement is around 0.5%. Considering structured (channel-wise) sparsity is one challenging sparsity type to achieve good accuracy, these results indicate S 2 -SAM works very well on structured sparse training methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.3">Improvement on the Roughness of the Loss Surface</head><p>In this part, we plot the loss surface of the sparse training with and without S 2 -SAM. The comparisons are demonstrated in Figure <ref type="figure">3</ref>, where we perform our experiments with three representative sparse training methods SNIP <ref type="bibr">[21]</ref>, GraSP <ref type="bibr">[18]</ref> and MEST <ref type="bibr">[3]</ref> at 90%, 95% and 98% sparsity ratios on  CIFAR-10 dataset with ResNet-32. We observe that as sparsity increases, suggesting more chaotic training behavior, the coefficient Ra also increases, indicating a steeper loss surface and a narrower basin. While with S 2 -SAM, we can see that under the same sparsity, the basin of the loss surface widens and enlarges as Ra values decrease, suggesting a smoother loss trajectory during training of a sparse network.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.4">Training Speed Comparison on GPU</head><p>S 2 -SAM is a highly efficient approach to optimizing the sharpness of the loss surface, which incurs zero extra cost for achieving its functionality. Compared to the traditional SAM, S 2 -SAM uses fewer computations and has better training performance on GPUs. In</p><p>Table 4, we obtain the computation cost as well as the training speed in terms of throughput (i.e., imgs/sec) on 4&#215; NVIDIA A6000 GPUs using ResNet-50 on the ImageNet-1K dataset. We record the throughput for different sparse training methods with their original training, with SAM [25], and with S 2 -SAM. From the results, we can see that S 2 -SAM achieves the same training throughput (with only negligible &lt; 20 imgs/s decrease due to processing) as the baseline methods, where no sharpness optimization is involved. Compared to original SAM, we can see that training throughput with SAM is less than half of the ones that train with S 2 -SAM, which aligns with the observation that SAM uses twice the computation cost of S 2 -SAM or original training. Although SAM yields slightly better accuracy, the associated training cost outweighs the benefits, rendering it impractical.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.5">Robustness Improvement by S 2 -SAM</head><p>Since sparse training approaches have potentially high usage in practical scenarios, the system's robustness against perturbation (e.g., inclement weather conditions for image/video tasks) is usually critical. We perform experiments to evaluate the sparse model (80% sparsity) robustness against perturbations in Table <ref type="table">5</ref>. Our intuition is that the model trained with its sharpness optimized has a wider loss basin, which indicates higher endurance on perturbations since the loss won't change much when it is located in a wide and flat region. We adopt ImageNet-C <ref type="bibr">[37]</ref> which contains a test set with the same images as ImageNet-1K but with nineteen types of corruptions applied with five different levels of severity. We train the sparse networks using different methods on ImageNet-1K, and then test their accuracy on ImageNet-C. We report test accuracy for both datasets. We can see that when the sparse model is trained using the original method, the accuracy of ImageNet-C is significantly lower than the accuracy of the original ImageNet-1K test set (around 35% lower). With S 2 -SAM, the test accuracy on ImageNet-C shows promising improvement. The robust accuracy improves by an average of 3.23%.  </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.6">Applying S 2 -SAM to Dense Model Training</head><p>We also apply S 2 -SAM to the dense model training. The results are shown in Table <ref type="table">6</ref>. We test on two datasets CIFAR-10 and ImageNet-1K with two additional networks MobileNet-V2 <ref type="bibr">[41]</ref> and EfficientNet-B0 <ref type="bibr">[42]</ref>. For ResNet-32 and VGG-19 on CIFAR-10, we follow the settings in <ref type="bibr">[3]</ref> and train for 160 epochs, and for MobileNet-V2, we use the 1.0 width version and train for 350 epochs for better convergence <ref type="bibr">[43]</ref>. For ResNet family on ImageNet-1K, we follow the setting in <ref type="bibr">[3]</ref> and train for 150 epochs, and we train EfficientNet-B0 for 350 epochs for better convergence <ref type="bibr">[44]</ref>. From the results, we can see that S 2 -SAM still improves the accuracy of the dense networks. We find out that the less the parameter count of the network, the better effectiveness of S 2 -SAM it achieves. This phenomenon proves our finding that a narrower network is harder to train, and S 2 -SAM is an effective solution for better generalization. We must also stress that S 2 -SAM is focusing on sparse neural network training, and we will leave the study of S</p><p>2 -SAM on dense model training for our future research. 4 Related Works Static Sparse Training Static sparse training determines the structure of the sparse network through the application of a pruning algorithm in the early stages of training. The lottery ticket hypothesis (LTH) <ref type="bibr">[20,</ref><ref type="bibr">45,</ref><ref type="bibr">46]</ref> uses iterative magnitude-based pruning (IMP) to find a subnetwork that can be trained from scratch without losing accuracy. SNIP <ref type="bibr">[21]</ref>, GraSP <ref type="bibr">[18]</ref>, SynFlow <ref type="bibr">[22]</ref> determine a static sparse pattern at training initialization by obtaining gradient information with a few iterations of dense training. FISH <ref type="bibr">[47]</ref> acquires fixed subnetwork by pre-computing a sparse mask using Fisher information.</p><p>Dynamic Sparse Training Dynamic sparse training starts with a randomly selected sparse network structure and adapts it throughout the training process in an effort to find a better sparse structure. Sparse Evolutionary Training (SET) <ref type="bibr">[2]</ref> prunes small magnitude weights and grows back randomly at the end of each training epoch. Deep R <ref type="bibr">[48]</ref> uses a combination of stochastic parameter updates and dynamic sparse parameterization for training. Dynamic Sparse Reparameterization (DSR) <ref type="bibr">[33]</ref> proposes to redistribute parameters between layers during training. Sparse Networks from Scratch (SNFS) <ref type="bibr">[49]</ref> creates the sparse momentum algorithm, which finds the layers and weights that effectively reduce the error by using exponentially smoothed gradients (momentum). In RigL <ref type="bibr">[23]</ref>, the gradients of all the weights are computed when the model needs to be updated to grow new connections. Top-KAST <ref type="bibr">[40]</ref> proposes a scalable and performant sparse-to-spars DST framework for maximum efficacy. Powerpropagation <ref type="bibr">[50]</ref> suggests a novel neural network weight parameterization that largely preserves low-magnitude parameters from learning. ITOP <ref type="bibr">[51]</ref> investigates the underlying DST mechanism and finds that the advantages of DST result from a time-based search for all potential factors. MEST <ref type="bibr">[3]</ref> designs a memory-economic sparse training framework targeting accurate and fast execution on edge devices. By co-training dense and sparse models, AD/AC <ref type="bibr">[52]</ref> suggests a technique that, at the conclusion of training, produces precise sparse-dense model pairings. Chase <ref type="bibr">[39]</ref> dynamically translates the unstructured sparsity into channel-level sparsity to achieve direct speedup on GPU.</p><p>Sharpness-Aware Minimization Sharpness-Aware Minimization was first introduced in <ref type="bibr">[25]</ref>. This optimization technique is designed to identify flatter minima characterized by consistently low loss in neighboring regions, aiming to enhance the model generalization capability during training. Nevertheless, the computational cost of Sharpness-Aware Minimization is doubled due to its two-step gradient computation regime, presenting challenges for deployment in large-scale training scenarios.</p><p>To reconcile such, ESAM <ref type="bibr">[26]</ref> introduces two novel and efficient training strategies: stochastic weight perturbation and sharpness-sensitive data selection, enhancing the efficiency of the SAM process without compromising its generalization performance. LookSAM <ref type="bibr">[27]</ref> only periodically calculates the inner gradient ascent to significantly reduce the additional training cost of SAM. SAF <ref type="bibr">[28]</ref> introduces a novel trajectory loss based on KL-divergence to measure the rate of change in training loss along the model update trajectory and replace the SAM sharpness measure. CrAM <ref type="bibr">[53]</ref> optimizes over the compression projection applied to the intermediate model at every training step, which results in a compressible models that can be pruned in an on-shot manner after training. However, all those methods need extra computation cost, and not target on sparse training method.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Conclusion</head><p>In this paper, we propose a novel Single-step Sharpness-Aware Minimization that is tailored for Sparse training (S 2 -SAM), which revolutionizes the originally computation-intensive sharpness-aware optimization into a highly efficient tool with zero extra cost. In light of the improved optimization trajectory in the loss surface, S 2 -SAM successfully enhances the accuracy of the sparse network training, as well as the robustness of the sparse model in practical scenarios. S 2 -SAM offers seamless plug-and-play functionality, showcasing its potential for widespread applicability in the evolving landscape of efficient training. The research is inherently scientific, and we anticipate no adverse societal impact stemming from its findings.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B Proof of Lemma 1</head><p>Proof. We consider two cases.</p><p>(</p><p>(</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C Proof of Theorem 1</head><p>Proof. By Theorem 3.2 of <ref type="bibr">[29]</ref>, we have</p><p>where (a) uses Lemma 1; (b) uses &#955; = 2L; (c) uses &#8710; t0 = 0; (d) uses &#951;L &lt; 1. Then, by Lemma 3.11 of <ref type="bibr">[29]</ref> and &#8467; is (L + &#955;)-smooth, we have</p><p>where the last inequality holds by selecting t 0 = 1 and &#955; = 2L. Then, by Theorem 2.2 of [29]</p><p>which is the generalization error. Next, we will bound the optimization error. Since F S is (L + &#955;)smooth and satisfies &#181;-PL condition, then follow the standard analysis, we have Since F S is (L + &#955;)-smooth, we have</p><p>where the last inequality is due to &#8467;(y, f (w, x)) is B-Lipschitz. Therefore, we get</p><p>Since F S satisfies &#181;-PL condition, then</p><p>Then by ( <ref type="formula">12</ref>), ( <ref type="formula">17</ref>) and ( <ref type="formula">21</ref>), we get</p><p>By using the conditions that min w&#8712;W F S (w) &#8804; F S (w * S ) + &#955; 2 &#8741;w t &#8741; 2 , definitions (11), we get</p><p>Therefore, we have the following inequality by ( <ref type="formula">22</ref>) and ( <ref type="formula">23</ref>):</p><p>By setting</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>NeurIPS Paper Checklist</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Claims</head><p>Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We propose S 2 -SAM which is tailored for sparse training. The proposed method is efficient with zero extra cost and achieves better accuracy on sparse training methods. We also demonstrate theoretical and experimental results in the paper. Guidelines:</p><p>&#8226; The answer NA means that the abstract and introduction do not include the claims made in the paper. &#8226; The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. &#8226; The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. &#8226; It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Limitations</head><p>Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The research is inherently scientific, and we anticipate no adverse societal impact stemming from its findings. We discuss the limitations in the experiments that our method is focusing on sparse training. But we also include dense training results to show potentials of our method. We will leave the dense model training of S 2 -SAM in our future research.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. &#8226; The authors are encouraged to create a separate "Limitations" section in their paper.</p><p>&#8226; The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. &#8226; The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. &#8226; The authors should reflect on the factors that influence the performance of the approach.</p><p>For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. &#8226; The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. &#8226; If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. &#8226; While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.</p><p>&#8226; The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). &#8226; The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) &#8226; The assumptions made should be given (e.g., Normally distributed errors). &#8226; It should be clear whether the error bar is the standard deviation or the standard error of the mean.</p><p>&#8226; It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. &#8226; For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). &#8226; If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments Compute Resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: We report the training resources we use in the experimental settings. Guidelines: &#8226; The answer NA means that the paper does not include experiments. &#8226; The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. &#8226; The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. &#8226; The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <ref type="url">https://neurips.cc/public/EthicsGuidelines</ref>? Answer: [Yes] Justification: Our research conforms with the NeurIPS Code of Ethics in every respect.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.</p><p>&#8226; If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. &#8226; The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="10.">Broader Impacts</head><p>Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?</p><p>Answer: [Yes] Justification: We discuss the broader impacts in the conclusion. The research is inherently scientific, and we anticipate no adverse societal impact stemming from its findings.</p><p>Guidelines:</p><p>&#8226; The answer NA means that there is no societal impact of the work performed.</p><p>&#8226; If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. &#8226; Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. &#8226; The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. &#8226; The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. &#8226; If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="11.">Safeguards</head><p>Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?</p><p>Answer: [NA] Justification: Our method doesn't have high risk for misuse.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper poses no such risks.</p><p>&#8226; Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. &#8226; Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. &#8226; We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.</p><p>12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?</p><p>Answer: [Yes] Justification: We credit every referenced assets with citations.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not use existing assets.</p><p>&#8226; The authors should cite the original paper that produced the code package or dataset.</p><p>&#8226; The authors should state which version of the asset is used and, if possible, include a URL. &#8226; The name of the license (e.g., CC-BY 4.0) should be included for each asset.</p><p>&#8226; For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.</p><p>&#8226; If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. &#8226; For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. &#8226; If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: This paper introduces new code for sharpness-aware minimization, and we will release the code after the paper is accepted. Guidelines:</p><p>&#8226; The answer NA means that the paper does not release new assets.</p><p>&#8226; Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. &#8226; The paper should discuss whether and how consent was obtained from people whose asset is used.</p><p>&#8226; At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and Research with Human Subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: This paper doesn't perform crowdsourcing and research with human subjects. Guidelines: &#8226; The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. &#8226; Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. &#8226; According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: This paper doesn't perform research with human subjects. Guidelines: &#8226; The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. &#8226; Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.</p><p>&#8226; We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. &#8226; For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.</p></div></body>
		</text>
</TEI>
