Introduction

Modern deep learning training frameworks heavily depend on large-scale datasets for achieving high accuracy.

This encourages deep learning practitioners to scrape data from the web for data collection [8,31,39,42]. Since a lot of the data is publicly available online, sometime this scrapping of data is unauthorized. For instance, a recent article [15] discloses that a private company trained a commercial face recognition system using over three billion facial images collected from the internet without any user consent. Although such massive data can significantly boost the performance of deep learning models, it raises serious concerns about data privacy and security.

To prevent the unauthorized usage of personal data, a series of recent papers [10,17,55] propose to poison data with additive noise. The idea is to make datasets unlearnable for deep learning models by ensuring that they learn the correspondence between noises and labels. Thereby, they do not learn much useful information about the clean data, significantly degrading their clean test accuracy. However, in recent works [11,17,49], these unlearnability methods are shown to be vulnerable to adversarial training (AT) frameworks [34]. Motivated by this problem, Fu et al. [11] developed Robust Error-Minimization (REM) noises to make unlearnable data that is protected from AT. While the authors show the effectiveness of REM in multiple scenarios, we demonstrate that these methods are still not robust against different data augmentations or training settings (see Section 3.2). Furthermore, current unlearnability frameworks [10,11,17,55] are model-dependent and require expensive optimization steps on deep learning models to obtain the additive noises. They also need to train the deep learning models from scratch to obtain noises for each new data set.

In this paper, we propose a novel Convolution-based Unlearnable DAtaset (CUDA) generation technique. We address limitations of existing unlearnable data generation techniques in Section 3.2 and motivate our CUDA technique in Section 3.3. For generating CUDA, an attacker randomly generates different convolutional filters for each class in the dataset using a private key or seed value. These filters are used to perform controlled class-wise convolutions on the clean training dataset to obtain CUDA. As we describe in Sections 3.3 and 5.2, CUDA generation performs controlled convolutions using a blur parameter p b to ensure that the semantics of the dataset are preserved (see Figure 1). CUDA generation with a lower blurring parameter p b adds less perceptible noises to clean samples. A network trained by a defender on CUDA is encouraged to learn the shortcut relation between class-wise convolutional filters and labels rather than useful features for classifying the clean data. Since the seed value for generating the filters are private, its not possible for the defender to obtain clean data from CUDA alone. Additionally, CUDA exhibits unlearnability effect with ERM even when only a fraction of the training dataset is perturbed (see Section 5). While the existing unlearnability works use additive noises, CUDA generation technique enjoys the advantage of introducing multiplicative noises in the Fourier domain due to the convolution theorem (since convolution of signals is the same as element-wise multiplication in the Fourier domain). This lets CUDA generation add higher amounts of noise in the image space, specifically along the edges in images, and makes it resilient to AT with small additive noise budgets. In Figure 2, with the help of t-SNE plots [51], we also find that the noises added by CUDA generation is not linearly separable while they are linearly separable for the existing works on unlearnability [54].

In Section 4, we theoretically show that CUDA generation can successfully poison Gaussian mixture data by degrading the clean data accuracy of the optimal Bayes classifier. We state our result informally below while the formal version is presented in Theorem 2.

Theorem 1 (Informal) Let D denote a Gaussian mixture data with two modes, P D denote the optimal Bayes classifier trained on D, and τ D (P D ) denote the accuracy of the classifier P D on D. Then, under some assumptions, for every clean data D, there is a CUDA D such that

Furthermore, our empirical experiments in Section 5 demonstrate the effectiveness of CUDA under various training scenarios such as ERM with various augmentations and regularizations, AT with different budgets, randomized smoothing [6,22,27], transfer learning, and fine-tuning. For instance, training a ResNet-18 on CIFAR-10 CUDA achieves only 18.48%, 44.4%, and 51.14% clean test accuracies with ERM, L ∞ AT, and L 2 AT, respectively (see Figure 2). Here, ERM on the clean training data achieves a clean test accuracy of 94.66%. In addition, we also design adaptive defenses to investigate if CUDA breaks with random or adversarial defense mechanisms. We find that CUDA is robust to the adaptive defenses that we specifically design to break it.

Related Works

Our CUDA generation technique is intimately related with adversarial and poisoning attacks. We first discuss some of this literature and then explain their relation with CUDA generation.

Adversarial attacks. Adversarial examples are specially designed examples that can fool deep learning models at test time [4,12,23,24,47]. The adversary crafts these examples by adding error-maximizing noises to the clean data. Even slightly perturbed data can serve as adversarial examples. AT is a training framework proposed to make deep learning models robust to adversarial examples [19,25,34,53,59]. AT is a min-max optimization problem where the model is trained to minimize loss on adversarial examples that have the maximum loss.

Poisoning attacks. In data poisoning, an attacker aims to hurt the deep learning model's performance by perturbing the training data [2,20,28,32,43,52]. The backdoor attack is a special type of poisoning attack where a trigger pattern is injected into clean data at training time [5,29,33,36]. The model trained on this data would misclassify an image with a trigger pattern at test time. Gu et al. [13] and Li et al. [30] use perceptible amounts of noises similar to CUDA for data poisoning. However, backdoor attacks do not affect the performance of the model on clean data [1,5,43].

Recent literature utilize data poisoning to protect data from being used for model training without authorization. Yuan and Wu [55] use neural tangent kernels [18] to generate clean label attacks that can hurt the generalization of deep learning models. Huang et al. [17] show that errorminimizing noise addition can serve as a poisoning technique. Fowl et al. [10] show that error-maximizing noises as well can make strong poison attacks. However, all these poisoning techniques do not offer data protection with AT [49,54]. Fu et al. [11] proposes a min-min-max optimization technique to generate poisoned data that offers better unlearnability effects with AT.

Convolution-based Ulearnable DAtaset (CUDA)

In this section, first we give some preliminaries about unlearnability. Then we discuss the limitations of the existing unlearnability methods. Finally, we propose our CUDA generation technique.

Preliminaries

Let {(x i , y i )} n i=1 ∼ D n be the clean training dataset where D is the clean data distribution, x i ∈ X ⊂ R d are the samples, and y i ∈ Y are the corresponding labels. Suppose a network is given as f θ : X → Y where θ ∈ Θ is the network parameter and : Y × Y → R is the loss function. ERM trains a network using a minimization problem of the form: min θ

where ρ a is the adversarial perturbation radius.

We will use τ D (θ), with θ the clean model parameter, to denote the clean test accuracy of a model trained on the clean training dataset (i.e., clean model accuracy). In unlearnable dataset generation, an attacker uses an algorithm A : X → X to generate an unlearnable dataset {( xi = A(x i ), y i )} n i=1 ∼ Dn from the clean training data. Here, the attacker assumes access to the full clean training datatset. Moreover, the attacker cannot modify the unlearnable dataset once it is released publicly. A defender trains using the unlearnable dataset to obtain a network f θ . The objective of the attacker is to design an unlearnable dataset such that the defender's model trained on the unlearnable data achieves a clean test accuracy (i.e., unlearnable model accuracy) worse than the clean model accuracy i.e. τ D ( θ) τ D (θ).

Limitations of existing works

Fu et al. [11] show that the previous unlearnability methods including Error-Minimization (EM) [17], Targeted Adversarial Poisoning (TAP) [10], and Neural Tangent Generalization Attack (NTGA) [55] are vulnerable to AT. Hence, they propose a Robust Error-Minimization (REM) [11] method that exploits a min-min-max optimization procedure to generate unlearnable noises. REM first trains a noise generator f θ on data points {(x i , y i )} n i=1 over a loss func-

Here, T is a distribution over a set of transformations {t : X → X }, ρ u is the defensive perturbation radius, and ρ a controls the protection level of REM against AT. After training the noise generator, an unlearnable example (x, y) is generated via

First, note that REM is computationally expensive since it needs to generate unlearnability noises through solving optimization equation 2. Moreover, the existing techniques are model-dependent and they require gradient-based training with a neural network to generate unlearnable data. They also require neural network training from scratch for every dataset that is to be made unlearnable. Table 1 shows the amount of time required to generate various unlearnable datasets using NVIDIA ® Tesla V100 GPU and 10 CPU cores. CUDA generation is significantly faster than the existing methods since it uses a model-free approach (no training required). Furthermore, REM is sensitive to hyperparameters and norm-budgets of AT since they generate noises with fixed L ∞ norm budgets. For instance, a ResNet-18 trained on clean CIFAR-10 dataset achieves a clean test data accuracy of 94.66%. L ∞ AT with perturbation radius ρ a = 4/255 on REM CIFAR-10 data (generated using ρ u = 8/255 and ρ a = 4/255) achieves only a clean test accuracy of 48.16%. However, L ∞ AT with perturbation radius ρ a = 8/255 and L 2 AT with perturbation radius ρ a = 0.75 can achieve a clean test accuracy of 78.71% and 79.65%, respectively, on the same REM data. We also find that ERM with a ResNet-18 on grayscaled REM CIFAR-10 images can achieve a high test accuracy of 70.76% on the grayscaled CIFAR-10 test data. This shows that REM relies upon the color space for poisoning clean data. Fu et al. [11] also show that REM noise generated using ResNet-18 is not transferable to DenseNet-121.

Our method: CUDA

The major limitations of the previous works are that they are vulnerable to AT, and computationally expensive. We think the major reason for the former limitation is the usage of small additive noises for unlearnability. AT is designed to train in the presence of such additive noises. Increasing the budget of the amount of additive noises for unlearnability might destroy the semantics of the images while perturbing them. The latter limitation arises from the fact that these methods are model-dependent and they require multilevel optimizations. Hence, we are motivated to design a compute-efficient unlearnability method that is robust to AT. CUDA technique can perform convolutions to add larger amounts of noises to clean images without destroying its semantics. This can help CUDA to be robust against AT. CUDA uses randomly generated convolution filters for blurring images from each class. This makes a model trained on CUDA to learn shortcut relations between filters and labels. We empirically support these claims in Section 5.2. Additionally, randomly generating the filters makes our technique model-free. Since the keys for generating the filters are private, it is not possible to reverse the blurring effect in CUDA without having access to the corresponding clean images. Hence, we assume that the data publisher deletes the clean images after perturbing them. Moreover, CUDA technique is a novel class of non-additive noise based poisoning attack that needs to be studied.

CUDA uses convolutional filters s i ∈ R k×k for each class i ∈ [1, K]. A random parameter, out of the k 2 parameters, in each of the filters is set to have a value of 1. The rest of the filter parameters are randomly initialized from a uniform distribution U (0, p b ) using a private seed where p b is the blur parameter. Blur parameter controls the level of blurring that occurs when an image x ∈ [0, 1] d1×d2×d3 is convolved with a filter s i . Here, d 1 , d 2 , and d 3 are the height, width, and number of channels of the image, respectively. For example, a CIFAR-10 image has a dimension of 32 × 32 × 3. The higher p b is, the higher the blurring effect is. Let x = x * s i where x belongs to class i. The CUDA data point for x is given by x = x/MAX( x). Rescaling is performed to make sure that the CUDA image pixels lie between 0 and 1. We find that the unlearnability effect gets stronger with larger p b and k values (see supplementary material for details).

Theory for CUDA

In this section, we define a binary classification setup similar to [19,35] to theoretically analyze CUDA. Let D be a clean dataset modelled by an isotropic Gaussian mixture model given by N (yµ, I), where y ∈ {±1} is the class label, µ ∈ R d , and I ∈ R d×d is the identity matrix. We defer the proofs for all the lemmas and theorem to the sup- To characterize the decision boundary for CUDA, we need to use some properties of Toeplitz matrices from Noschese et al. [37] given in the following Lemma 3.

Lemma 3 Any tri-diagonal Toeplitz matrix

Next we use Lemma 3 to show that the Bayes optimal decision boundary for classifying D is a quadratic plane.

). The Bayes optimal decision boundary for classifying D is given by P (x) ≡ x Ax + b x + c = 0, where

Now we state a lemma regarding the tail of Gaussian quadratic forms which plays a crucial role in our main result.

Lemma 5 Let • denote the operator norm, • 2 denote the vector 2-norm, z ∼ N (0, I), and Z = z Az +z b+c where A = QΛQ . Using Chernoff bound, for any t ≥ 0 and γ ∈ R,

.

Lemma 5 allows us to provide an upper bound for the accuracy of the unlearnable decision boundary P on the clean dataset D, given as τ D ( P ), in Theorem 2 below.

Theorem 2 (Main result)

For any non-negative constants t 1 and t 2 , the accuracy of the unlearnable decision boundary P on the clean dataset D can be upper-bounded as

Moreover, for any µ = 0 and a -1 ∈ [0, 0.5], ∃a 1 such that τ D ( P ) < τ D (P ).

Poisoning is effective only if the accuracy of the unlearnable model P is less than that of the clean model P on the clean dataset D, that is, τ D ( P ) < τ D (P ). To satisfy this condition, we need to carefully select a y 's. In Theorem 2, we formally state this condition 1 . Theorem 2 shows that CUDA can effectively poison when there are two distinct modes in the clean Gaussian mixture data model. We validate our theoretical claim through empirical analysis as well (see Figure 3). Details for the analysis is given in supplementary material. We find that our upper bound for the clean test accuracy of CUDA classifiers are consistent with our empirical analysis. In our experiments, we also find that the unlearnability effect is stronger with a larger blur parameter. This effect is evident from Figure 3 where we find that it is likely to get a lower τ D ( P ) with higher a y values. These results are consistent with our experimental results in Section 5 with CIFAR-10, CIFAR-100, and ImageNet-100 datasets.

Experiments

In this section, we first discuss our experimental setup. More details on the setup is deferred to supplementary material. We then show the robustness of CUDA generation with various datasets and architectures. We also run various experiments to analyze the effectiveness of CUDA under different training techniques (ERM, AT with varying budgets, randomized smoothing, transfer learning, and finetuning) and augmentation techniques (mixup [58], grayscaling, random blurring, cutout [9], cutmix [56], autoaugment [7], and orthogonal regularization [3]). Finally, we also design adaptive defenses to test the robustness of CUDA. One might think that CUDA filters can be obtained by adversarially training them with the data. We show that CUDA is robust to such adaptive defenses that we design.

Experimental setup

Datasets. We use three image classification datasets -CIFAR-10, CIFAR-100, [21] and ImageNet-100 (a subset of ImageNet made of the first 100 classes) [40]. We use the data augmentation techniques such as random flipping, cropping, and resizing [44].

Architectures. We use ResNet-18 [14], VGG-16 [45], Wide ResNet-34-10 [57], and DenseNet-121 [16]. We train the networks with hyperparameters used in Fu et al. [11]. Previous works mainly employ ResNet-18 for most of their evaluations. Additional experiments on Tiny-ImageNet [26], DeIT [50], EfficientNetV2-S [48], and MobileNetV2 [41] are provided in the supplementary material.

CUDA generation. We use filters of size k = 3 and blur parameter p b = 0.3 for both CIFAR-10 and CIFAR-100 datsets. ImageNet-100 is a higher dimensional 224×224×3 image dataset when compared to the 32×32×3 dimensional CIFAR datasets. Hence, we use larger filters of size k = 9 with p b = 0.06 for ImageNet-100. These hyperparameters are chosen such that the CUDA images are not perceptibly highly perturbed and give good unlearnability effect (see plot in Figure 1). In supplementary matrial, we show the results of training on CUDA with different hyperparameters for data generation.

Baselines. We compare CUDA generation technique with four state-of-the-art unlearnability methods -REM [11], EM [17], TAP [10], and NTGA [55]. We adopt the results reported in [11] since we use the same hyperparameters for training. For REM, we select hyperparameters ρ u = 8/255 and ρ a = 4/255, the highest radii values in [11]. For comparing unlearnable methods, we look at the clean test accuracy. The lower the test accuracy, the better the unlearnability method is. As mentioned in the supplementary material, we use publicly released official codebases for reproducing the baselines using their default hyperparameters.

Effectiveness of CUDA

Why does CUDA work? In order to measure how much CUDA technique's blurring affects the dataset's quality, we compare class-wise blurring (CUDA technique) against uni- versal blurring where a single convolutional filter is used for blurring all the images. We keep the filter generation parameters fixed (p b = 0.3 and k = 3) for both CUDA classwise blurring and universal blurring. ResNet-18 trained with clean CIFAR-10, universally blurred CIFAR-10, and CIFAR-10 CUDA achieve clean test accuracies of 94.66%, 90.47%, and 18.48%, respectively. This suggests that our controlled blurring does not obscure the semantics of the dataset. Hence, the significant drop in the clean test accuracy introduced by CUDA is most likely due to the usage of class-wise filters. This suggests that a model trained on CUDA learns the relation between the class-wise convolution filters and their corresponding labels. Therefore, during test time when this convolution effect is absent, the CUDA trained model fails to make correct classifications. Furthermore, the model trained on CUDA achieves an accuracy of 99.91% on the CIFAR-10 CUDA testset. This strongly supports our claim that the CUDA model learns to classify images based on the convolutional filters used to blur them.

In addition, if we permute the class-wise filters for blurring the test set (i.e., blurring class 1 images with class 2 filters, class 2 images with class 3 filters, and so on), we get a very low accuracy of 2.53% on this test set. Further details are provided in the supplementary material. Finally, we note that real-world datasets might also contain blurred images due to various factors such as motion blurring, weather conditions, issues with the camera, etc. Hence, detecting if a blurred image is poisoned might not always be possible. However, one might argue that it is possible to detect if an entire dataset is blurred. Interestingly, later in this section we show that CUDA technique exhibits unlearnability effect even when only a fraction of the training dataset is poisoned. Different datasets. We first compare the effectiveness of CUDA with different datasets using ERM and L ∞ AT with ρ a = 4/255. We use ResNet-18 for the experiments. The results are shown in Table 2. These results show that EM, TAP, and NTGA are not robust to AT. However, both CUDA and REM are successful. Here, our method CUDA outperforms REM with CIFAR-10 and ImageNet-100 datasets. Smartly designed additive noise in AT helps in achieving better generalization than ERM on the unlearnable datasets. This experiment thus demonstrates that ERM and AT are not good choices for training with CUDA and REM dataset.

Different models. Next we compare the effectiveness of CUDA using various deep learning architectures with L ∞ AT (ρ a = 4/255). We use CIFAR-10 for the experiments. The results are shown in Table 3. As we see in the table, CUDA is effective with all the five network architectures. However, REM is not seen to be transferrable with DenseNet-121.

Robustness to different training settings. In Section 3.2, we show that REM is sensitive to the training settings. REM generated using L ∞ radii budgets of ρ u = 8/255 and ρ a = 4/255 for CIFAR-10 breaks with L ∞ AT (ρ a = 8/255) and L 2 AT (ρ a = 0.75) to get test accuracy of 78.71% and 79.65%, respectively. Hence, we run experiments to check the robustness of CUDA with various AT norm budgets. The results are shown in Table 4. As we see in the table, CUDA is robust to ERM, L ∞ , and L 2 AT settings with varying training budgets. Impressively, the highest test accuracy achieved with training on CIFAR-10 CUDA, CIFAR-100 CUDA, and ImageNet-100 CUDA are as low as 51.19%, 36.90%, and 40.08%, respectively. We also find that using a pre-trained ResNet-18 with CIFAR-10 CUDA only achieves clean test accuracy of 42.42% and 48.22% with fine-tuning the full network and a newly trained final layer, respectively (details are deferred to the supplementary material).

Different protection percentages. In Section 3.1, we assume that the attacker has access to the full clean training data. However, in real life settings, this might not be always possible. Hence, we train ResNet-18 on a mix of CIFAR-10 CUDA and clean CIFAR-10 training datasets to evaluate the effectiveness of poisoning with varying data protection percentages. Protection percentage denotes the percentage of the training data that is poisoned.

We show the results in Table 5. In the table, the "Mixed" column denotes the clean test accuracy of a model trained One may think that CUDA technique can be broken by learning the private filters from the data. We test this idea by training deconvolution filters to find if we can reverse the blurring effect in CUDA with adversarially trained filters. We use a novel Deconvolution-based Adversarial Training (DAT) technique that is similar to AT (check supplementary material for details). While the adversarial step in AT learns sample-wise error-maximizing additive noises, the adversarial step in DAT learns class-wise error-maximizing deconvolution filters. We train DAT with filters of varying sizes (3, 5, and 7) on CIFAR-10 CUDA using ResNet-18. The filter parameters are constrained within a finite range to make sure that the images do not get distorted with the adversarial step similar to projection in projected gradient descent. We find that CUDA is robust to DAT. DAT using filters of size 3, 5, and 7 with CUDA achieves only test accuracy of 39.05%, 46.21%, and 38.48%, respectively. DAT is not successful against CUDA since we can not invert convolutions without the knowledge of the private filters or clean images corresponding to CUDA.

Limitations and future directions. The unlearnability effect of CUDA can be defended if some fraction of the clean data and its corresponding CUDA images are leaked. The defender must also be able to detect if all samples in the dataset are poisoned. However, our work assumes a setup where the filters remain private. For example, a data publisher could simply publish their CUDA images and delete the clean images permanently to prevent this scenario. As discussed in this section, CUDA as well as other prior works do not perform well with different protection percentages with AT. Improving this can be an interesting research direction. We believe that extending CUDA technique to other domains such as tabular and text data is also an interesting future direction. It would also be interesting to see theoretical analysis of CUDA considering more complex setups.

[58] Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin At the optimal decision boundary the probabilities of any point x ∈ R d belonging to class y = -1 and y = 1 modeled by D are the same. Here, µ = µ 1 = -µ -1 and

Now, the accuracy of the clean model P is to be computed. Note that if P (x) < 0 the Bayes optimal classification is class -1, else the classification is class 1. Let z ∼ N (0, I), and Z ∼ N (0, 1), and sgn(.) be the signum function.

A.2. Proof for Lemma 2

Let D 1 = N (µ, I). For every data point (x, y) ∼ D 1 , let the perturbed data (A 1 x, y) be modelled by a distribution D1 . We prove that D1 = N (A 1 µ, A 1 A 1 ).

Tri-diagonal Toeplitz matrices A y = T (d; a y , 1, a y ) are symmetric. Hence, D = N (yA y µ, A 2 y ).

A.3. Remarks on Lemma 3

A tri-diagonal Toeplitz matrix T (d; a 1 , a 2 , a 3 ) is represented as

The class of matrices A y = T (d; a y , 1, a y ) are symmetric and can be diagonalized as QDQ .

1/2 sin ijπ d+1 i,j is symmetric and it is the common eigenvector matrix to all A y matrices. As shown in Lemma 3, Q and D can be represented using trigonometric functions. Also, we have

A.4. Proof for Lemma 4

At the optimal decision boundary the probabilities of any point x ∈ R d belonging to class y = -1 and y = 1 modeled by D are the same. Here, µ = µ 1 = -µ -1 and A y 's are symmetric.

2 )

Note that here if P (x) < 0, the Bayes optimal classification is class -1, else the classification is class 1. Here, for shorthand notations we denote

A.5. Proof for Lemma 5

Let Z = z Az +z b+c and z ∼ N (0, I) ⊂ R d where A = QΛQ . Also,

For any t ≥ 0 and x ∼ N (0, I), we write the moment generating function for a quadratic random variable Y = x Ax as 2

.

Using the Chernoff bound and E z Az = Tr(AE[zz ]) = Tr(A), for some γ,

since Λ is a diagonal matrix. Using Woodbury matrix identity, we get (I -2tΛ) -1 = I -(I -1 2t Λ -1 ) -1 . This gives us

.

A.6. Proof for Theorem 2

Note that if P (x) < 0, the classifier predicts a label for class -1, else the predicted label would be 1. Here, x = yµ + z where z ∼ N (0, I) and y ∈ {±1} since (x, y) ∼ D.

We can see that

Using Lemma 5, with γ = γ 1 , t = t 1 for computing p 1 and γ = γ 2 , t = t 2 for computing p 2 where t 1 , t 2 are some non-negative constants, we get

, and

This gives us the upper bound for τ D ( P ). However, to make sure that this upper bound is smaller than 1, we need to assert more conditions. p 1 and p 2 become smaller as γ 1 and γ 2 are larger positive numbers. However,

) 0. Hence, we look at separately at cases when either γ 1 > 0 or γ 2 > 0.

If γ 1 > 0, then τ D ( P ) = 1 2 (p 1 + 1) < 1. Else, if γ 2 > 0, then τ D ( P ) = 1 2 (p 2 + 1) < 1. We know that for µ = 0, τ D (P ) = φ(µ) > 1 2 . Moreover, for any a -1 ∈ [0, 0.5], ∃a 1 such that τ D ( P ) < τ D (P ). This can be satisfied by picking a 1 such that either γ 1 or γ 2 is very large, i.e., 1 2 < τ D ( P ) = A.7. Details on generating Figure 3 We use µ ∈ R d , d = 100 to generate clean dataset with 1000 data points. They are randomly split into training and testing partitions of equal size. All the assumptions are consistent with the details provided in the main body. We use 30 × 30 mesh-grid to plot the contour plots. While plotting the theoretical upper bounds, we choose the best t 1 , t 2 with grid search from a search space

A.8. Experimental details

This subsection provides the details for experiments in Section 5.

Hardware. We use NVIDIA ® RTX A4000 GPU with 16GB memory with 16 AMD ® EPYC 7302P CPU cores.

Data augmentations. For CIFAR-10 and CIFAR-100, we use random flipping, 4 pixel padding, and random 32 × 32 size cropping. For ImageNet-100, we use random flipping and random cropping with resizing to 224 × 224 size. All the images are rescaled to have pixel values in the range [0, 1].

Baselines. We compare CUDA against error-minimizing noise [17], targeted adversarial poisoning [10], neural tangent generalization attack [55], and robust error-minimizing noise [11]. We use the experimental outputs reported in [11] for our comparisons. For REM we choose ρ u = 8/255 and ρ a = 4/255 since REM works the best when ρ u = 2ρ a [11]. We perform experiments on REM not present in their work using their code available publicly on GitHub 3 (MIT License).

Networks. For consistency, we use the same architectures used in [11]. We use their GitHub script 4 for this purpose.

Training. We train all the networks for 100 epochs. The initial learning rate is 0.1. Learning rate decays to 0.01 at epoch 40 and to 0.001 at epoch 80. We use a stochastic gradient descent optimizer with a momentum factor of 0.9, weight decay factor of 0.0005, and batch size of 128. For adversarial training, we follow the procedure in [34]. We use 10 steps of projected gradient descent with a step size of 0.15ρ a .

3 https://github.com/fshp971/robust-unlearnableexamples 4 https://github.com/fshp971/robust-unlearnableexamples/tree/main/models Analysis of CUDA. For grayscaling experiments, we use images with their average channel values as the input to the network. Test accuracy is computed on the grayscaled test datasets. For smoothing, we use the GitHub codesfoot_2 from [6] (MIT License). For mixup [58], we use the default value of α = 1.0.

Deconvolution-based adversarial training (DAT). We experiment with various filter sizes of 3,5, and 7 for the transpose convolution filters. For each batch of data, we use 10 steps of projected gradient descent with a learning rate of 0.1 to learn transpose convolution filters for each class. The weights and biases of the transpose convolution filters are constrained to be within [-C, C]. We choose C = 5. After 10 steps of inner maximizing optimization, the resulting image is rescaled such that the pixel values lie in [0, 1]. See Figure 4 for clean test accuracy vs. epochs plot for DAT with varying transpose filter sizes. As seen in Figure 1, DAT can break CUDA CIFAR-10 with a low blur parameter value of p b = 0.1 to get a clean test accuracy ∼78%. However, with higher p b values DAT can not achieve more than 50% clean test accuracy. DAT solves the following optimization problem:

where denotes the transpose convolution operator, s yi denotes the transpose convolution filter for class y i , and is the soft-max cross-entropy loss function.

CUDA with augmentations. We use mixup with the default α = 1.0 [58]. See Figure 5 for the training curve. For random blurring augmentations, we use p b = 0.1, 0.3 and k = 3. With both these parameters, CUDA is seen to be effective. See Figure 5 for the training curve with p b = 0.3.

A.9. More experimental results

Figure 6 shows the CUDA CIFAR-10 data generated using k = 3 and different p b blur parameters. Figure 7 shows the CUDA CIFAR-100 and CUDA ImageNet-100 data generated using k = 3, p b = 0.3 and k = 9, p b = 0.06, respectively. Figure 8 shows the clean test accuracy of ResNet-18 with CUDA CIFAR-10 generated using different blur parameters. As we see in the plots, higher the blur parameter, better the effectiveness of CUDA is. However, we choose p b = 0.3 for our experiments since the the images generated using this hyperparameter look perceptibly more similar to the clean images (when compared to p b = 0.5) while giving a very low clean test accuracy. A lower value of p b = 0.1 gives better unlearnability. However, CUDA CIFAR-10 generated using p b = 0.1 is not robust with We show the effectiveness of CUDA with Tiny-ImageNet [26], DeIT [50], EfficientNetV2 [48], and Mo-bileNetV2 [41] below. (see Figure 11). This experiment also demonstrates that the blurring we perform does not make the dataset useless or destroy its semantics. For this experiment, we use models with fixed initialization and random seeds.

A.11. Why does CUDA work?

In this section, we perform experiments that show that a model trained on CUDA dataset learns the relation between the class-wise filters and the labels. We train ResNet-18 using the CUDA CIFAR-10 dataset for the experiments. We perform three independent trials for each of the experiments and report the mean performance scores. Trained models achieve a mean clean test accuracy of 21.34%. Now, we use the class-wise filters to perturb the images in the test set based on their corresponding labels. Trained models achieve a very high mean accuracy of 99.91% on this perturbed test set. This shows that the trained models learned the relation between the filters and their corresponding labels. Next, we permute the filters to perturb the test set such that test set images with label i are perturbed with the filters of class (i + 1)%10. Trained models achieve a very low mean accuracy of 2.53% on this perturbed test set. This is evidence that CUDA can also be used for backdoor attacks.

A.12. Effect of transfer learning

In this section, we experiment the effect of using a pretrained ResNet-18 with PyTorch [38]. We train it on the CUDA CIFAR-10 dataset in two different ways. First, we fine-tune the whole network on the CUDA dataset with a learning rate of 0.001 for 15 epochs. This achieves a clean test accuracy of 42.42%. Fine-tuning the network with clean training data gives 94.19% clean test accuracy. Next, we freeze all the layers except the final layer to train a linear classifier with the pre-trained weights using the CUDA CIFAR-10 dataset. We call this "Freeze and learn". We use a SGD optimizer to train the linear layer for 15 epochs with an initial learning rate of 0.1. The learning rate is decayed by a factor of 10 after every 5 epochs. This achieves a clean test accuracy of 48.22%. The results are shown in Figure 12. This experiment shows that pre-trained network with CUDA data training does not help achieve good generalization on the clean data distribution.

A.13. Effect of CUDA with regularization techniques

In this section, we study the effect of training a ResNet-18 with CUDA CIFAR-10 dataset using various regularization techniques such as mixup [58], cutout [9], cutmix [56], autoaugment [7], and orthogonal regularization [3]. We perform mixup, cutout, cutmix, autoaugment, and orthogonal regularization to achieve 25.53%, 25.80%, 26.93%, 34.09%, and 50.72%. Even though these regularizations help in improving the vanilla ERM training, these networks still do not achieve good generalization on the clean data distribution. We use cutout using GitHub codes 6with length=16 and n holes=1, cutmix using GitHub codes 7 with α = 1, autoaugment using PyTorch [38], mixup using GitHub codesfoot_5 with α = 1, and orthogonal regularization using GitHub codesfoot_6 with reg=1e-6 (all MIT licenses).

A.14. Network parameter distribution

In this section, we compare the network parameter distributions of ResNet-18 trained on clean and CUDA CIFAR-10 datasets (see Figure 13). Both the distributions are similar to normal distributions with a mean of 0. However, the parameter distribution of the clean model has a higher standard deviation than the CUDA-based model's parameter distribution.