A central theme of machine learning is to alleviate the issue of overfitting, improving the generalization performance on testing data. This is often achieved by leveraging important prior knowledge of the models and data of interest. For example, the regularization-based methods introduce penalty on the complexity of the model, which often amount to enforcing certain smoothness properties. Data augmentation techniques, on the other hand, leverage important invariance properties of the data (such as the shift and rotation invariance of images) to improve performance. Novel approaches that exploit important knowledge of the models and data hold the potential of substantially improving the performance of machine learning systems.
We propose MaxUp, a simple yet powerful training method to improve the generalization performance and alleviate the over-fitting issue. Different from standard methods that minimize the average risk on the observed data, MaxUp generates a set of random perturbations or transforms of each observed data point, and minimizes the average risk of the worst augmented data of each data point. This allows us to enforce robustness against the random perturbations and transforms, and hence improve the generalization performance. MaxUp can easily leverage arbitrary stateof-the-art data augmentation schemes (e.g. Zhang et al., 2018;DeVries & Taylor, 2017;Cubuk et al., 2019a), and substantially improves over them by minimizing the worst (instead of average) risks on the augmented data, without adding significant computational ahead.
Theoretically, in the case of Gaussian perturbation, we show that MaxUp effectively introduces a gradient-norm regularization term that serves to encourage smoothness of the loss function, which does not appear in standard data augmentation methods that minimize the average risk.
MaxUp can be viewed as a "lightweight" variant of adversarial training against adversarial input pertubrations (e.g. Tramèr et al., 2018;Madry et al., 2017), but is mainly designed to improve the generalization on the clean data, instead of robustness on perturbed data (although MaxUp does also increase the adversarial robustness in Gaussian adversarial certification as we shown in our experiments (Section 4.4)). In addition, compared with standard adversarial training methods such as projected gradient descent (PGD) (Madry et al., 2017), MaxUp is much simpler and computationally much faster, and can be easily adapted to increase various robustness defined by the corresponding data augmentation schemes.
We test MaxUp on three challenging tasks: image classification, language modeling, and certified defense against adversarial examples (Cohen et al., 2019). We find that MaxUp can leverage the different state-of-the-art data augmentation methods and boost their performance to achieve new state-of-the-art on a range of tasks, datasets, and neural architectures. In particular, we set up a new state-of-the-art result on ImageNet classification without extra data, which improves the best 85.5% top1 accuracy by Xie et al. (2019) to 85.8%. For the adversarial certification task, we find Maxup allows us to train more verifiably robust classifiers than prior arts such as the PGD-based adversarial training proposed by Salman et al. (2019).
We start with introducing the main idea of MaxUp, and then discuss its effect of introducing smoothness regularization in Section 2.1.
ERM Giving a dataset D n = {x i } n i=1 , learning often reduces to a form of empirical risk minimization (ERM):
where θ is a parameter of interest (e.g., the weights of a neural network), and L(x, θ) denotes the loss associated with data point x. A key issue of ERM is the risk of overfitting, especially when the data information is insufficient.
We propose MaxUp to alleviate overfitting. The idea is to generate a set of random augmented data and minimize the maximum loss over the augmented data.
Formally, for each data point x in D n , we generate a set of perturbed data points {x ′ i } m i=1 that are similar to x, and estimate θ by minimizing the maximum loss over {x ′ i }:
This loss can be easily minimized with stochastic gradient descent (SGD). Note that the gradient of the maximum loss is simply the gradient of the worst copy, that is,
where i * = arg max i∈[m] L(x ′ i , θ). This yields a simple and practical algorithm shown in Algorithm 1.
In our work, we assume the augmented data {x ′ i } m i=1 is i.i.d. generated from a distribution P(•|x). The P(•|x) can be based on small perturbations around x, e.g., P(•|x) = N (x, σ 2 I), the Gaussian distribution with mean x and isotropic variance σ 2 . The P(•|x) can also be constructed based on invariant data transformations that are widely used in the data augmentation literature, such as random crops, equalizing, rotations, and clips for images (see e.g Cubuk et al., 2019a;DeVries & Taylor, 2017;Cubuk et al., 2019b).
We provide a theoretical interpretation of Maxup as introducing a gradient-norm regularization to the original ERM objective to encourage smoothness. Here we consider the simple case of isotropic Gaussian perturbation, when P(•|x) = N (x, σ 2 I). For simplifying notation, we define
which represents the expected MaxUp risk of data point x with m augmented copies.
Theorem 1 (MaxUp as Gradient-Norm Regularization).
Consider LP,m (x, θ) defined in (4) with
where c m,σ is a constant and c m,σ = Θ(σ √ log m), where Θ(•) denotes the big-Theta notation.
Theorem 1 shows that, the expected MaxUp risk can be viewed as introducing a Lipschitz-like regularization with the gradient norm ∇ x L(x, θ) 2 , which encourages the smoothness of L(x, θ) w.r.t. the input x. The strength of the regularization is controlled by c m,σ , which depends on the number of samples m and perturbation magnitude σ.
Proof. Using Taylor expansion, we have
where we assume z i = x ′ ix, which follows N (0, σ 2 I). The rest of the proof is due to the Lemma 1 below.
Lemma 1. Let g be a fixed vector in R d , and {z i } m i=1 are m i.i.d. random variables from N (0, σ 2 I). We have
which is well known to be Θ(σ √ log m). See e.g., Orabona & Pál (2015); Kamath (2015)
MaxUp is closely related to both data augmentation and adversarial training. It can be viewed as an adversarial variant of data augmentation, in that it minimizes the worse case loss on the perturbed data, instead of an average loss like typical data augmentation methods. MaxUp can also be viewed as a "lightweight" variant of adversarial training, in that the maximum loss is calculated by simple random sampling, instead of more accurate gradient-based optimizers for finding the adversarial loss, such as projected gradient descent (PGD); MaxUp is much simpler and faster than the PGD-based adversarial training, and is more suitable for our purpose of alleviating over-fitting on clean data (instead of adversarial defense). We now elaborate on these connections in depth.
Data augmentation has been widely used in machine learning, especially on image data which admits a rich set of invariance transforms (e.g. translation, rotation, random cropping). Recent augmentation techniques, such as MixUp (Zhang et al., 2018), CutMix (Yun et al., 2019) and manifold MixUp (Verma et al., 2019) have been found highly useful in training deep neural networks, especially in achieving state-of-the-art results on important image classification benchmarks such as SVHN, CIFAR and Im-ageNet. More recently, more advanced methods have been developed to find the optimal data augmentation policies using reinforcement learning or adversarial generative network (e.g. Cubuk et al., 2019a;b;Zhang et al., 2020).
MaxUp can easily leverage these advanced data augmentation techniques to achieve good performance. The key difference, however, is that MaxUp in (2) minimizes the maximum loss on the augmented data, while typical data augmentation methods minimize the average loss, that is,
which we refer to as standard data augmentation through-out the paper. It turns out ( 2) and ( 5) behave very different as regularization mechanisms, in that ( 5) does not introduce the gradient-norm regularization as (2), and hence does not have the benefit of having gradient-norm regularization. This is because the first-order term in the Taylor expansion is canceled out due to the averaging in (5).
Specifically, let P(•|x) be any distribution whose expectation is x and L(x, θ) is second-order differentiable w.r.t x.
Define the expected loss related to (5) on data point x:
Then with a simple Taylor expansion, we have
which misses the gradient-norm regularization term when compared with MaxUp decomposition in Theorem 1.
Note that the MaxUp update is computationally faster than the solving (5) with the same m, because we only need to backpropagate on the worst augmented copy for each data point (see Equation 3), while solving (5) requires to backpropagate on all the m copies at each iteration.
Adversarial training has been developed to defense various adversarial attacks on the data inputs (Madry et al., 2017). It estimates θ by solving the following problem:
where B(x, r) represents a ball centered at x with radius r under some metrics (e.g. ℓ 0 , ℓ 1 , ℓ 2 , or ℓ ∞ distances). The inner maximization is often solved by running projected gradient descent (PGD) for a number of iterations.
MaxUp in (2) can be roughly viewed as solving the inner adversarial maximization problem in (7) using a "mild", or "lightweight" optimizer by randomly drawing m points from P(•|x) and finding the best. Such mild adversarial optimization increases the robustness against the random perturbation it introduces, and hence enhance the generalization performance. Adversarial ideas have also been used to improvement generalization in a series of recent works (e.g., Xie et al., 2019;Zhu et al., 2020).
Different from our method, typical adversarial training methods, especially these based PGD (Madry et al., 2017), tend to solve the adversarial optimization much more aggressively to achieve higher robustness, but at the cost of scarifying the accuracy on clean data. There has been shown a clear trade-off between the accuracy of a classifier on clean data and its robustness against adversarial attacks (see e.g., Tsipras et al., 2019;Zhang et al., 2019;Yin et al., 2019;Schmidt et al., 2018). By using a mild adversarial optimizer, MaxUp strikes a better balance between the accuracy on clean data and adversarial robustness.
Besides, MaxUp is much more computationally efficient than PGD-based adversarial training, because it does not introduce additional back-propagation steps as PGD. In practice, MaxUp can be equipped with various complex data augmentation methods (in which case P(•|x) can be discrete distributions), while PGD-based adversarial training mostly focuses on perturbations in ℓ p balls.
Online hard example mining (OHEM) (Shrivastava et al., 2016) is a training method originally developed for regionbased objective detection, which improves the performance of neural networks by picking the hardest examples within mini batches of stochastic gradient descent (SGD). It can be viewed as running SGD for minimizing the following expected loss
which amounts to randomly picking a mini-batch M at each iteration and minimizing the loss of the hardest example within M. By doing so, OHEM can focus more on the hard examples and hence improves the performance on borderline cases. This makes OHEM particularly useful for class-imbalance tasks, e.g. object detection (Shrivastava et al., 2016), person reidentification (Luo et al., 2019).
Different with MaxUp, the hardest examples in OHEM are selected in mini-batches consisting of independently selected examples, with no special correlation or similarity. Mathematically, it can be viewed as reweighing the data distribution to emphasize harder instances. This is substantially different from MaxUp, which is designed to enforce the robustness against existing random data augmentation/perturbation schemes.
Top-1 error Top-5 error Vanilla (He et al., 2016b) 76.3 -Dropout (Srivastava et al., 2014) 76.8 93.4 DropPath (Larsson et al., 2017) 77.1 93.5 Manifold Mixup (Verma et al., 2019) 77.5 93.8 AutoAugment (Cubuk et al., 2019a) 77.6 93.8 Mixup (Zhang et al., 2018) 77.9 93.9 DropBlock (Ghiasi et al., 2018) 78.3 94.1 CutMix (Yun et al., 2019) 78.6 94.0 MaxUp+CutMix 78.9 94.2
Table 1. Summary of top1 and top5 accuracies on the validation set of ImageNet for ResNet-50.
We test our method using both image classification and language modeling for which a variety of strong regularization techniques and data augmentation methods have been proposed. We show that MaxUp can outperform all of these methods on the most challenging datasets (e.g. ImageNet, Penn Treebank, and Wikitext-2) and state-of-the-art models (e.g. ResNet, EfficientNet, AWD-LSTM). In addition, we apply our method to adversarial certification via Gaussian smoothing (Cohen et al., 2019), for which we find that MaxUp can outperform both the augmented data baseline and PGD-based adversarial training baseline.
For all the tasks, if training from scratch, we first train the model with standard data augmentation with 5 epochs and then switch to MaxUp.
Cost MaxUp only slightly increase the time and memory cost compared with standard training.
During MaxUp, we only need to find the worst instance out of the m augmented copies through forward-propagation, and then only back-propagate on the worst instance. Therefore, the additional cost of MaxUp over standard training is m forward-propagation, which introduces no significant overhead on both memory and time cost.
We evaluate MaxUp on ILSVRC2012, a subset of Im-ageNet classification dataset (Deng et al., 2009). This dataset contains around 1.3 million training images and 50,000 validation images. We follow the standard data processing pipeline including scale and aspect ratio distortions, random crops, and horizontal flips in training. During the evaluation, we only use the single-crop setting. CutMix randomly cuts and pasts patches among training images, while the ground truth labels are also mixed proportionally to the area of the patches. MaxUp+CutMix applies CutMix on one image for m times (cutting different randomly sampled patches), and select the worst case to do backpropagation.
We test our method on ResNet-50, ResNet-101 (He et al., 2016b), as well as recent energy-efficient architectures, including ProxylessNet (Cai et al., 2019) and Efficient-Net (Tan & Le, 2019). We resize the images to 600 × 600 and 845 × 845 for EfficientNet-B7 and EfficientNet-B8, respectively (Tan & Le, 2019), for which we process the images with the data processing pipelines proposed by Touvron et al. (2019). For the other models, the input image size is 224 × 224. To save computation resources, we only fine-tune the pre-trained models with MaxUp for a few epochs. We set m = 4 for MaxUp in the ImageNet-2012 experiments unless indicated otherwise. This means that we optimize the worst case in 4 augmented samples for each image.
For ResNet-50, ResNet-101 and ProxylessNets, we train the models for 20 epochs with learning rate 10 -5 and batch size 256 on 4 GPUs for 20 epochs. For EfficientNet, we fix the parameters in the batch normalization layers and train the other parameters with learning rate 10 -4 and batch size 1000 for 5 epochs.
We test MaxUp equipped with Cutout (DeVries & Taylor, 2017) on CIFAR-10 and CIFAR-100, and denote it by MaxUp+Cutout. We conduct our method on several neural architectures, including ResNet-110 (He et al., 2016b), PreAct-ResNet-110 (He et al., 2016a) and WideResNet-28-10 (Zagoruyko & Komodakis, 2016). We set m = 10 for WideResNet and m = 4 for the other models. We use the public code 2 and keep their hyper-parameters.
Implementation Details For CIFAR-10 and CIFAR-100, we use the standard data processing pipeline (mirror+ crop) and train the model with 200 epochs. All the results reported in this section are averaged over five runs.
We starts at 0.1 and is divided by 10 after 100 and 150 epochs for ResNet-110 and PreAct-ResNet-110. For WideResNet-28-10, we follow the settings in the original paper (Zagoruyko & Komodakis, 2016), where the learning rate is divided by 10 after 60, 120 and 180 epochs.
Weight decay is set to 2.5 -4 for all the models, and we do not use dropout.
The results on CIFAR-10 and CIFAR-100 are summarized in Table 3 andTable 4. We can see that the models trained using MaxUp+Cutout significantly outperform the standard Cutout for all the cases.
On CIAFR-10, MaxUp improves the standard Cutout baseline from 94.84% ± 0.11% to 95.41% ± 0.08% on ResNet-110. It also improves the accuracy from 95.02% ± 0.15% to 95.52% ± 0.06% on PreAct-ResNet-110.
On CIFAR-100, MaxUp obtains improvements by a large margin. On ResNet-110 and PreAct-ResNet-110, MaxUp improves the performance of Cutout from 73.64% ± 0.15% and 74.37% ± 0.13% to 75.26% ± 0.21% and 75.63% ± 0.26%, respectively. MaxUp+Cutout also improves the standard Cutout from 81.59% ± 0.27% to 82.48% ± 0.23% on WideResNet-28-10 on CIFAR-100.
Ablation Study We test MaxUp with different sample size m and investigate its impact on the performance on ResNet-100 (a relatively small model) and WideResNet-28-10 (a larger model).
Table 5 shows the result when we vary the sample size in m ∈ {1, 4, 10, 20}. Note that MaxUp reduces to the naïve data augmentation method when m = 1. As shown in Table 5, MaxUp with all m > 1 can improve the result of standard augmentation (m = 1). Setting m = 4 or m = 10 achieves best performance on ResNet-110 , and m = 10 obtains best performance on WideResNet-28-10. We can see that the results are not sensitive once m is in a proper range (e.g., m ∈ [4 : 10]), and it is easy to outperform the standard data augmentation (m = 1) without much tuning of m. Furthermore, we suggest to use a large m for large models, and a small m for relatively small models.
For language modeling, we test MaxUp on two benchmark datasets: Penn Treebank (PTB) and Wikitext-2 (WT2). We use the code provided by Wang et al. (2019) as our baselinefoot_1 , which stacks a three-layer LSTM and implements a bag of regularization and optimization tricks for neural language modeling proposed by Merity et al. (2018), such as weight tying, word embedding drop and Averaged SGD.
For this task, we apply MaxUp using word embedding dropout (Merity et al., 2018) as the random data augmentation method. Word embedding dropout implements dropout on the embedding matrix at the word level, where the dropout is broadcasted across all the embeddings of all the word vectors. For the selected words, their embedding vectors are set to be zero vectors. The other word embeddings in the vocabulary are scaled by 1 1-p , where p is the probability of embedding dropout.
As the word embedding layer serves as the first layer in a neural language model, we apply MaxUp in this layer. We do feed-forward for m times and select the worst case to do backpropagation for each given sentence. In this section, we set a small m = 2 since the models are already wellregularized by other regularization techniques.
Implement Details The PTB corpus (Marcus et al., 1993) is a standard dataset for benchmarking language models. It consists of 923k training, 73k validation and 82k test words. We use the processed version provided by Mikolov et al. (2010) that is widely used for PTB.
The WT2 dataset is introduced in Merity et al. (2018) as an alternative to PTB. It contains pre-processed Wikipedia articles, and the training set contains 2 million words.
The training procedure can be decoupled into two stages: 1) optimizing the model with SGD and averaged SGD (ASGD); 2) restarting ASGD for fine-tuning twice. We apply MaxUp in both stages, and report the perplexity scores at the end of the second stage. We also report the perplexity scores with a recently-proposed post-process method, dy- namical evaluation (Krause et al., 2018) after the training process.
Results on PTB and WT2 The results on PTB and WT2 corpus are illustrated in Table 6 andTable 7, respectively. We calculate the perplexity on the validation and test set for each method to evaluate its performance. We can see that MaxUp outperforms the state-of-the-art results achieved by Frage (Gong et al., 2018) and Mixture of SoftMax (Yang et al., 2018). We further compare MaxUp to the result of Wang et al. (2019) based on AWD-LSTM (Merity et al., 2018) at two checkpoints, with or without dynamic evaluation (Krause et al., 2018)
Modern image classifiers are known to be sensitive to small, adversarially-chosen perturbations on inputs (Goodfellow et al., 2014). Therefore, for making high-stakes decisions, it is of critical importance to develop methods with certified robustness, which provide (high probability) provable guarantees on the correctness of the prediction subject to arbitrary attacks within certain perturbation ball.
Recently, Cohen et al. (2019) Training Details We applied MaxUp to Gaussian augmented data on CIFAR-10 with ResNet-110 (He et al., 2016b). We follow the training pipelines described in Salman et al. (2019). We set a batch size of 256, an initial learning rate of 0.1 which drops by a factor of 10 every 50 epochs, and train the models for 150 epochs.
Evaluation After training the smoothed classifiers, we evaluation the certified accuracy of different models under different ℓ 2 perturbation sets. Given an input image x and a perturbation region B, the smoothed classifier is called certifiably correct if its prediction is correct and has a guaranteed lower bound larger than 0.5 in B. The certified accuracy is the percentage of images that are certifiably correct. Following Salman et al. (2019), we calculate the certified accuracy of all the classifiers for various radius and report the best results overall of the classifiers. We use the codes provided by Cohen et al. (2019) to calculate certified accuracy. 4
Following Salman et al. (2019), we select the best hyperparameters with grid search. The only two hyperparameters of our MaxUp+Gauss are the sample size m and the variance σ 2 of the Gaussian perturbation, which we search in m ∈ {5, 25, 50, 100, 150} and σ ∈ {0.12, 0.25, 0.5, 1.0}.
In comparison, Salman et al. (2019) requiers to search a larger number of hyper-parameters, including the number of steps of the PGD, the number of noise samples, the maximum ℓ 2 perturbation, and the variance of Gaussian data augmentation during training and testing. Overall, Salman et al. (2019) requires to train and evaluate over 150 models for hyperparmeter tuning, while MaxUp+Gauss requires only 20 models.
We show the certified accuraries on CIFAR-10 in Table 8 under ℓ 2 attacks for each ℓ 2 radius. We find that MaxUp outperforms Cohen et al. (2019) for all the ℓ 2 radiuses by a large margin. For example, MaxUp can im-4 https://github.com/locuslab/smoothing prove the certified accuracy at radius 0.25 from 60% to 74% and improve the 4% accuracy on radius 2.75 to 15%.
MaxUp also outperforms the PGD-based adversarial training of Salman et al. (2019) for all the radiuses, boosting the accuracy from 14% to 17% at radius 2.5, and from 12% to 15% at radius 2.75.
In summary, MaxUp clearly outperforms both Cohen et al. (2019) and Salman et al. (2019). MaxUp is also much faster and requires less hyperparameter tuning than Salman et al. (2019). Although the PGD-based method of Salman et al. (2019) was designed to outperform the original method by Cohen et al. (2019), MaxUp+Gauss further improves upon Salman et al. (2019), likely because MaxUp with Gaussian perturbation is more compatible with the Gaussian smoothing based certification of Cohen et al. (2019) than PGD adversarial optimization.
In this paper, we propose MaxUp, a simple and efficient training algorithms for improving generalization, especially for deep neural networks. MaxUp can be viewed as a introducing a gradient-norm smoothness regularization for Gaussian perturbation, but does not require to evaluate the gradient norm explicitly, and can be easily combined with any existing data augmentation methods. We empirically show that MaxUp can improve the performance of data augmentation methods in image classification, language modeling, and certified defense. Especially, we achieve SOTA performance on ImageNet.
For future works, we will apply MaxUp to more applications and models, such as BERT (Devlin et al., 2019). Furthermore, we will generalize MaxUp to apply mild adversarial optimization on feature and label spaces for other challenging tasks in machine learning, including transfer learning, semi-supervised learning.
All the FLOPS and model size reported in this paper is calculated by https://pypi.org/project/ptflops.
https://github.com/ChengyueGongR/advsoft