Bebis, G.
(Ed.)
In the last decade, convolutional neural networks (CNNs) have evolved to become the dominant models for various computer vision tasks, but they cannot be deployed in low-memory devices due to its high memory requirement and computational cost. One popular, straightforward approach to compressing CNNs is network slimming, which imposes an ℓ1 penalty on the channel-associated scaling factors in the batch normalization layers during training. In this way, channels with low scaling factors are identified to be insignificant and are pruned in the models. In this paper, we propose replacing the ℓ1 penalty with the ℓp and transformed ℓ1 (T ℓ1 ) penalties since these nonconvex penalties outperformed ℓ1 in yielding sparser satisfactory solutions in various compressed sensing problems. In our numerical experiments, we demonstrate network slimming with ℓp and T ℓ1 penalties on VGGNet and Densenet trained on CIFAR 10/100. The results demonstrate that the nonconvex penalties compress CNNs better than ℓ1 . In addition, T ℓ1 preserves the model accuracy after channel pruning, and ℓ1/2,3/4 yield compressed models with similar accuracies as ℓ1 after retraining.
more »
« less