Lifelong learning (L2M) is crucial for machine learning (ML) to acquire new skills through continual learning, pushing ML toward a more human learning in reality. Given a stream of different tasks and data, a deep neural network (DNN) can quickly learn a new task, by leveraging the acquired knowledge after learning previous tasks, under constraints in terms of the amount of computing and memory required (Chaudhry et al., 2019). As a result, it is quite challenging to train an L2M model with a high utility. Orthogonal to this, L2M models are vulnerable to adversarial attacks, i.e., privacy model attacks (Shokri et al., 2017;Fredrikson et al., 2015;Wang et al., 2015;Papernot et al., 2016), when DNNs are trained on highly sensitive data, e.g., clinical records (Choi et al., 2017;Miotto et al., 2016), user profiles (Roumia & Steinhubl, 2014;Wu et al., 2010), and medical images (Plis et al., 2014;Helmstaedter et al., 2013).
In practice, the privacy risk will be more significant since an adversary can observe multiple versions of an L2M model released after training on each task. Different versions of the model parameters can be considered as an additional information leakage, compared with a model trained on a single task (Theorem 1). Memorizing previous tasks while learning new tasks further exposes private information in the training set, by continuously accessing the data from the previously learned tasks (i.e., data stored in an episodic memory (Chaudhry et al., 2019;Riemer et al., 2019;Tao et al., 2020)); or accessing adversarial examples produced from generative memories to imitate real examples of past tasks (Shin et al., 2017;Wu et al., 2018;Ostapenko et al., 2019). Unfortunately, there is a lack of study offering privacy protection to the training data in L2M.
Our Contributions. To address this problem, we propose to preserve differential privacy (DP) (Dwork et al., 2006), a rigorous formulation of privacy in probabilistic terms, in L2M. We introduce a new definition of lifelong differential privacy (Lifelong DP), in which the participation of any data tuple in any tasks is protected under a consistently bounded DP guarantee, given the released parameters in both learning new tasks and memorizing previous tasks (Definition 3). This is significant by allowing us to train and release new versions of an L2M model, given a stream of tasks and data, under DP protection.
Based upon this, we propose a novel L2DP-ML algorithm to preserve Lifelong DP. In L2DP-ML, privacy-preserving noise is injected into inputs and hidden layers to achieve DP in learning private model parameters in each task (Alg. 1). Then, we configure the episodic memory as a stream of fixed and disjoint batches of data, to efficiently achieve Lifelong DP (Theorem 2). The previous task memorizing constraint is solved, by inheriting the recipe of the wellknown A-gem algorithm (Chaudhry et al., 2019), under Lifelong DP. To our knowledge, our study establishes a formal connection between DP preservation and L2M given a growing number of learning tasks compared with existing works (Farquhar & Gal, 2018;Phan et al., 2019a). Rigorous experiments, conducted on permuted MNIST (Kirkpatrick et al., 2017), permuted CIFAR-10 datasets, and an L2M task on our collected dataset for human activity recognition in the wild show promising results in preserving DP in L2M.
Let us first revisit L2M with A-gem and DP. In L2M, we learn a sequence of tasks T = {t 1 , . . . , t m } one by one, such that the learning of each new task will not forget the models learned for the previous tasks. Let D i be the dataset of the i-th task. Each tuple contains data x ∈ [-1, 1] d and a ground-truth label y ∈ Z K , which is a one-hot vector of K categorical outcomes y = {y 1 , . . . , y K }. A single true class label y x ∈ y given x is assigned to only one of the K categories. All the training sets D i are non-overlapping; that is, an arbitrary input (x, y) belongs to only one D i , i.e., ∃!i ∈ [1, m] : (x, y) ∈ D i (x ∈ D i for simplicity). On input x and parameters θ, a model outputs class scores f : R d → R K that map inputs x to a vector of scores f (x) = {f 1 (x), . . . , f K (x)} s.t. ∀k ∈ [1, K] : f k (x) ∈ [0, 1] and K k=1 f k (x) = 1. The class with the highest score is selected as the predicted label for x, denoted as y(x) = max k∈K f k (x). A loss function L(f (θ, x), y) presents the penalty for mismatching between the predicted values f (θ, x) and original values y.
Lifelong Learning. Given the current task τ (≤ m), let us denote T τ = {t 1 , . . . , t τ -1 } is a set of tasks that have been learnt. Although there are different L2M settings, i.e., episodic memory (Rebuffi et al., 2017;Lopez-Paz & Ranzato, 2017;Riemer et al., 2019;Abati et al., 2020;Tao et al., 2020;Rajasegaran et al., 2020;Ebrahimi et al., 2020) and generative memory (Shin et al., 2017;Wu et al., 2018;Ostapenko et al., 2019), we leverage one of the state-of-the-art algorithms, i.e., A-gem (Chaudhry et al., 2019), to demonstrate our privacy-preserving mechanism, without loss of the generality of our study. A-gem avoids catastrophic forgetting by storing an episodic memory M i for each task t i ∈ T τ . When minimizing the loss on the current task τ , a typical approach is to treat the losses on the episodic memories of tasks i < τ , given by L(f (θ, M i )) = 1 |Mi| x∈Mi L(f (θ, x), y), as inequality constraints. In A-gem, the L2M objective function is:
where θ τ -1 are the values of model parameters θ learned after training the task
Eq. 1 indicates that learning θ τ given the task τ will not forget previously learned tasks {t 1 , . . . , t τ -1 } enforced by the memory replaying constraint
At each training step, A-gem (Chaudhry et al., 2019) has access to only D τ and M τ to compute the projected gradient g (i.e., by addressing the constraint in Eq. 1), as follows:
where g is the updated gradient computed on a batch sampled from D τ , g ref is an episodic gradient computed on a batch sampled from M τ , and g is used to update the model parameters θ in Eq. 1.
Differential Privacy (DP). DP guarantees that the released statistical results, computed from the underlying sensitive data, is insensitive to the presence or absence of one tuple in a dataset. Let us briefly revisit the definition of DP, as:
Definition 1 (ϵ, δ)-DP (Dwork et al., 2006). A randomized algorithm A is (ϵ, δ)-DP, if for any two neighboring databases D and D ′ differing at most one tuple, and ∀O ⊆ Range(A), we have:
where ϵ controls the amount by which the distributions induced by D and D ′ may differ, and δ is a broken probability.
A smaller ϵ enforces a stronger privacy guarantee.
DP has been preserved in many ML models and tasks (Abadi et al., 2017;Phan et al., 2019b;Papernot et al., 2018). However, existing mechanisms have not been designed to preserve DP in L2M under a fixed and consistently bounded privacy budget given a growing stream of learning tasks. That differs from our goal in this study.
In this section, we focus on analyzing the unknown privacy risk in L2M and introduce a new concept of Lifelong DP.
Privacy Risk Analysis. One benefit of L2M is that end-users can use an L2M model after training each task τ , instead of waiting for the model to be trained on all the tasks. Thus, in practice, the adversary can observe the model parameters θ 1 , .
i differ at most one tuple; and (2) M i and M ′ i differ at most one tuple.
A Naive Mechanism. To preserve DP in L2M, one can employ the moments accountant (Abadi et al., 2016) to train the model f by injecting Gaussian noise into clipped gradients g and g ref (Eq. 2), with privacy budgets ϵ Dτ and ϵ Mτ on each dataset D τ and on the episodic memory M τ , and a gradient clipping bound C. The post-processing property in DP (Dwork et al., 2014) can be applied to guarantee that g, computed from the perturbed g and g ref , is also DP.
Let us denote this mechanism as A, and denote A τ as A applied on the task τ . A naive approach (Desai et al., 2021) is to repeatedly apply A on the sequence of tasks T. Since training data is non-overlapping among tasks, the parallel composition property in DP (Dwork & Lei, 2009) can be applied to estimate the total privacy budget consumed across all the tasks, as follows:
where ϵ = max i∈[1,m] (ϵ Di + ϵ Mi ), and ∀i, j ∈ [1, m] : δ is the same for ϵ Di and ϵ Mj .
A(data m ) indicates that the model is trained from scratch with the mechanism A, given randomly initiated parameters θ 0 , i.e., A(θ 0 , data m ). Intuitively, we can achieve the traditional DP guarantee in L2M, as the participation of a particular data tuple in each dataset D τ is protected under the released (ϵ, δ)-DP {θ i } i∈ [1,m] . However, this approach introduces unknown privacy risks in each task and in the whole training process, as discussed next.
Observing the intermediate parameters {θ i } i<τ turns the mechanism A τ into a list of adaptive DP mechanisms A 1 , . . . , A τ sequentially applied on tasks t 1 , . . . , t τ , where A i : (
This is an instance of adaptive composition, which we can model by using the output of all the previous mechanisms {θ i } i<τ as the auxiliary input of the A τ mechanism. Thus, given an outcome θ τ , the privacy loss c(•) at θ τ can be measured as follows:
The privacy loss is accumulated across tasks, as follows:
As a result of the Theorem 1, the privacy budget at each task τ cannot be simply bounded by max τ ∈[1,m] (ϵ Dτ + ϵ Mτ ), given δ (Eq. 4). This problem might be addressed by replacing the max function in Eq. 4 with a summation function:
, to compute the upper bound of the privacy budget for an entire of the continual learning process. To optimize this naive approach, one can adapt the management policy (Lécuyer et al., 2019) to redistribute the privacy budget across tasks while limiting the total privacy budget ϵ to be smaller than a predefined upper bound, that is, the training will be terminated when ϵ reaches the predefined upper bound.
However, the challenge in bounding the privacy risk is still the same, centering around the growing number of tasks m and the heterogeneity among tasks: (1) The larger the number of tasks, the larger the privacy budget will be consumed by the function. It is hard to identify an upper bound privacy budget given an unlimited number of streaming tasks in L2M; (2) Different tasks may require different numbers of training steps due to the difference in terms of the number of tuples in each task; thus, affecting the privacy budget ϵ; and (3) The order of training tasks also affect the privacy budget, since computing g ref by using data in the episodic memory from one task may be more than other tasks. Therefore, bounding the DP budget in L2M is non-trivial.
Lifelong DP. To address these challenges, we propose a new definition of ϵ-Lifelong DP to guarantee that an adversary cannot infer whether a data tuple is in the lifelong training dataset data m , given the released parameters {θ i } i∈ [1,m] learned from a growing stream of an infinite number of new tasks, denoted ∀m ∈ [1, ∞), under a consistently bounded DP budget ϵ (Eq. 6). A consistently bounded DP means having only one fixed value of ϵ, regardless of the number of tasks m. In other words, it does not exist an i ≤ m and an ϵ ′ < ϵ, such that releasing {θ j } j∈ [1,i] given training dataset data i is ϵ ′ -DP (Eq. 7). A consistently bounded DP is significant by enabling us to keep training and releasing an L2M model without intensifying the end-to-end privacy budget consumption. Lifelong DP can be formulated as follows:
Definition 3 ϵ-Lifelong DP. Given a lifelong database data m , a randomized algorithm A achieves ϵ-Lifelong DP, if for any of two lifelong neighboring databases (data m , data ′ m ), for all possible outputs
) where Range(A) denotes every possible output of A.
In our Lifelong DP definition, the episodic memory (data) M can be an empty set ∅ in the definition of lifelong neighboring databases (Def. 2) given L2M mechanisms that do not need to access M (Yoon et al., 2020;Ye & Bors, 2020;Maschler et al., 2021;Qu et al., 2021;He & Zhu, 2022).
To preserve Lifelong DP, we need to address the following problems: (1) The privacy loss accumulation across tasks;
(2) The overlapping between the episodic memory M and the training data D; and (3) The data sampling process for computing the episodic gradient g ref given the growing episodic memory M. The root cause issue of these problems is that in an L2M model, the episodic memory M, which accumulatively stores data from all of the previous tasks, is read at each training step. Thus, using the moments account to preserve Lifelong DP will cause the privacy budget accumulated, resulting in a loose privacy protection given a large number of tasks or training steps. Therefore, designing a mechanism to preserve Lifelong DP under a tight privacy budget is non-trivial and an open problem.
Algorithm 1 L2DP-ML Algorithm
To overcome the aforementioned issues, our idea is designing a L2M mechanism such that the privacy budget will not accumulate across training steps while memorizing previously learned tasks. More precisely, we design our network as a multi-layer neural network stacked on top of a feature representation learning model. Then, we propose a new Laplace mechanism-based Lifelong DP algorithm, called L2DP-ML (Alg. 1), in computing the gradients g, g ref , and g. Finally, to overcome expensive computation cost and heterogeneity among tasks, we develop a scalable and heterogeneous algorithm through a streaming batch training (Alg. 2), to efficiently learn Lifelong DP parameters (Theorem 2).
Network Design. In our Alg. 1 and Fig. 1, a DNN is designed as a stack of an auto-encoder for feature representation learning and a typical multi-layer neural network, as follows: f (x) = G(a(x, θ 1 ), θ 2 ) where a(•) is the auto-encoder and G(•) is the multi-layer neural network. The auto-encoder a(•) takes x as an input with model parameters θ 1 ; meanwhile, the multi-layer neural network G(•) takes the output of the auto-encoder a(•) as its input with model parameters θ 2 and returns the class scores f (x).
This network design allows us to: (1) Tighten the sensitivity of our model, since it is easy to train the auto-encoder using less sensitive objective functions, given its small sizes;
(2) Reduce the privacy budget consumption, since the computations of the multi-layer neural network is DP when the output of the auto-encoder is DP; and (3) Provide a better re-usability, given that the auto-encoder can be reused and shared for different predictive models.
Given a dataset D τ , the objective functions of the auto-encoder and the multi-layer neural network can be the classical cross-entropy error functions for data reconstruction at the input layer and for classification at the output layer, denoted R Dτ (θ 1 ) and L Dτ (θ 2 ) respectively. Without loss of generality, we define the data reconstruction function R Dτ (θ 1 ) and the classification function L Dτ (θ 2 ) as follows:
where the transformation of
is the reconstruction of x r , and h πr computed from the x r through the network with W π is the parameter at the last hidden layer h π .
Our L2M objective function is defined as:
where {θ 1 , θ 2 } are the model parameters; while, {θ τ 1 , θ τ 2 } are the values of {θ 1 , θ 2 } after learning task τ . At each training step on the current task τ , to update the model parameters {θ τ 1 , θ τ 2 } minimizing Eq. 10, we need to compute the gradients g and g ref , and then follow Eq. 2 to compute the projected gradient g for the model parameters {θ τ 1 , θ τ 2 } (Fig. 2). Given the projected g, we can update {θ τ 1 , θ τ 2 } by applying typical descent operation, as follows.
Gradient Update g. To compute the gradient g for {θ τ 1 , θ τ 2 } on the current task τ , we first derive polynomial forms of R Dτ (θ 1 ) and L Dτ (θ 2 ), by applying the 1st and 2nd orders of Taylor Expansion (Arfken, 1985) as follows:
To preserve ϵ 1 -DP in learning θ 1 , we leverage Functional Mechanism (Zhang et al., 2012) to inject a Laplace noise into polynomial coefficients of the function R Dτ (θ 1 ), which are the input x and the first transformation h 1 . Laplace mechanism (Dwork et al., 2014) is well-known in perturbing objective functions to prevent privacy budget accumulation in training ML models (Phan et al., 2017a;b;2020). As in (Phan et al., 2020), the global sensitivity ∆ R is bounded as follows:
where
and ϵ 1 is a privacy budget.
Importantly, the perturbation of each example x turns the original data (Phan et al., 2020) (Alg. 1, line 2). Based upon that, all the computations on top of the (ϵ 1 /γ x )-DP dataset D τ , including h r , h r , x r , and the computation of gradients g of the model parameters θ 1 are shown to be (ϵ 1 /γ x )-DP without accessing any additional information from the original data D τ , i.e., ∀s ∈ [1, d] :
. This follows the post-processing property of DP (Dwork et al., 2014). Consequently, the total privacy budget used to perturb R is (ϵ 1 + ϵ 1 /γ x ), by having
A similar approach is applied to perturb the objective function L Dτ (θ 2 ) at the output layer with a privacy budget ϵ 2 . The perturbed function of L is denoted as L Dτ (θ 2 ). As in Lemma 3 (Phan et al., 2020), the output of the auto-encoder, which is the perturbed transformation
and ∥θ 1 ∥ 1,1 is the maximum 1-norm of θ 1 's columnsfoot_0 . As a result, the computations of all the hidden layers of the multi-layer neural network G(•) that takes the output of the auto-encoder h 1Dτ as its input, is (ϵ 1 /γ)-DP, since h 1Dτ is (ϵ 1 /γ)-DP, following the post-processing property of DP (Dwork et al., 2014)
That helps us to (1) avoid extra privacy budget consumption in computing the multi-layer neural network G(•); (2) tighten the sensitivity of the function L Dτ (i.e., ∆ L ≤ 2|h π |); and (3) achieve DP gradient update for θ 2 . The total privacy budget used to perturb L is
Episodic and Projected Gradients g ref and g. Now, we are ready to present our approach in achieving Lifelong DP, by configuring the episodic memory at the current task τ (i.e., M τ ) as a fixed and disjoint set of datasets from previous tasks, i.e., M τ = {D 1 , . . . , D τ -1 } (Alg. 1, line 7); such that, at each training step, the computation of episodic gradients g ref for the model parameters {θ 1 , θ 2 } using a randomly picked dataset D ref ∈ M τ (Alg. 1, lines 8 and 11), is (ϵ 1 + ϵ 1 /γ x + ϵ 1 /γ + ϵ 2 )-DP, without incurring any additional privacy budget consumption for the dataset D ref . The projected gradients g is computed from g and g ref (Eq. 2) is also (ϵ 1 + ϵ 1 /γ x + ϵ 1 /γ + ϵ 2 )-DP, following the post-processing property of DP (Dwork et al., 2014).
Hence, we reformulate the L2M objective function in Eq. 10, as follows:
where
By using the perturbed functions R and L, the constrained optimization of Eq. 14 can be addressed similarly to Eq. 2, when the projected gradient g is computed as:
where g is the gradient update on the current task τ , and g ref is computed using a dataset D ref randomly selected from the episodic memory M τ .
Lifelong DP Guarantee. Given the aforementioned network f (x) as the stack of the auto-encoder and the multilayer neural network, and privacy budgets ϵ 1 and ϵ 2 , the total Lifelong DP privacy consumption in learning the model parameters {θ 1 , θ 2 } at each task is computed in Theorem 2.
Theorem 2 shows that Alg. 1 achieves ϵ-Lifelong DP in learning the model parameters at each task
, where ϵ = (ϵ 1 + ϵ 1 /γ x + ϵ 1 /γ + ϵ 2 ). There are three key properties in the proof of Theorem 2 (Appx. B):
(1) For every input x in the whole training set
x is included in one and only one dataset, denoted D x ∈ D (Eq. 17). Hence, the DP guarantee to x in D is equivalent to the DP guarantee to x in D x (Eqs. 19 and 20).
(2) If we randomly sample tuples from the episodic memory to compute the episodic gradients g ref , the sampling set and its neighboring set can have at most i -1 different tuples (i ∈ [1, m]), since each D i and its neighboring dataset D ′ i can have at most 1 different tuple. In addition, a random sampling set of tuples in the episodic memory can overlap with more than one datasets D i , which is used to compute the gradient g. Importantly, different sampling sets from the episodic memory can overlap each other; thus, a simple data tuple potentially is used in multiple DPpreserving objective functions using these overlapping sets to compute the episodic gradients g ref . These issues introduce additional privacy risk by following the group privacy theory and overlapping datasets in DP. We address this problem, by having the episodic memory as a fixed and disjoint set of datasets across T training tasks (Eq. 18). As a result, we can prevent the additional privacy leakage, caused by: (i) Differing at most i -1 tuples between neighboring M i and M ′ i for all i ∈ (1, m]; and (ii) Generating new and overlapping sets of data samples for computing the episodic gradient (which are considered overlapping datasets in the parlance of DP) in the typical training. Thus, the optimization on one task does not affect the DP protection of any other tasks, even the objective function given one task can be different from the objective function given other tasks (Eq. 21).
(3) Together with the results achieved in (1) and ( 2), by having one and only one privacy budget for every task, we can achieve Eqs. 6 and 7 in Lifelong DP (Def. 3). We present these steps in Eqs. 27 and 29.
Although computing the gradients given the whole dataset D τ achieves Lifelong DP, it has some shortcomings: (1) consumes a large computational memory to store the episodic memory; (2) computational efficiency is low, since we need to use the whole dataset D τ and D ref to compute the gradient update and the episodic gradient at each step; This results in a slow convergence speed and poor utility.
Scalability. To address this, we propose a streaming batch training (Alg. 2, Appx. C), in which a batch of data is used to train the model at each training step, by the following steps.
(1) Slitting the private training data D τ (∀τ ∈ T) into disjoint and fixed batches (Alg. 2, line 4).
(2) Using a single draw of Laplace noise across batches (Alg. 2, lines 1-2). That prevents additional privacy leakage, caused by: (i) Generating multiple draws of noise (i.e., equivalent to applying one DP-preserving mechanism multiple times on the same dataset); (ii) Generating new and overlapping batches (which are considered overlapping datasets in the parlance of DP); and (iii) More importantly, for any example x, x is included in only one batch. Hence, each disjoint batch of data in Alg. 2 can be considered as a separate dataset in Alg. 1.
(3) For each task, we randomly select a batch to place in the episodic memory (Alg. 2, line 17).
(4) At each training step, a batch from the current task is used to compute the gradient g, and a batch randomly selected from the episodic memory is used to compute the episodic gradient g ref (Alg. 2,. Thus, Alg. 2 still preserves (ϵ 1 + ϵ 1 /γ x + ϵ 1 /γ + ϵ 2 )-Lifelong DP (Theorem 2).
By doing so, we significantly reduce the computational complexity and memory consumption, since only a small batch of data from each task is stored in the episodic memory.
Heterogeneity. Based upon this, our algorithm can be applied to address the heterogeneity in terms of data sizes among tasks, which differs from multi-modal tasks (Liu et al., 2019). We can train one task with multiple epochs, without affecting the Lifelong DP protection in Alg. 2, by 1) keeping all the batches fixed among epochs, and 2) at the end of training each task, we randomly select a batch of that task to place in the episodic memory. The order of the task does not affect the Lifelong DP, since the privacy budget is not accumulated across tasks. These distinct properties enable us to customize our training, by having different numbers of training epochs for different tasks and having different training orders of tasks. Tasks with smaller numbers of data tuples can have larger numbers of training epochs. This helps us to achieve better model utility under the same privacy protection as shown in our experiments.
Our validation focuses on understanding the impacts of the privacy budget ϵ and the heterogeneity on model utility. For reproducibility, our implementation is available and uploaded.
Baseline Approaches. We consider A-gem (Chaudhry et al., 2019) as an upper bound in terms of model performance, since A-gem is a noiseless model. We aim to show how much model utility is compromised for the Lifelong DP protection. Also, we consider the naive algorithm (Desai et al., 2021), called NaiveGaussian, as a baseline to demonstrate the effectiveness of our L2DP-ML mechanism. It is worth noting that there is a lack of a precise definition of adjacent databases resulting in an unclear or not well-justified DP protection for L2M in existing works (Farquhar & Gal, 2018;Phan et al., 2019a). Therefore, we do not consider them as baselines in our experiments.
To evaluate the heterogeneity, we further derive several versions of our algorithm (Alg. 2), including: (1) Balanced L2DP-ML, in which all the tasks have the same number of training steps, given a fixed batch size. This is also true for a Balanced A-gem algorithm;
(2) L2DP-ML with the same number of epochs for all the tasks; and (3) Heterogeneous L2DP-ML, in which a fixed number of training epochs is assigned to each task. The numbers of epochs among tasks can be different. For instance, 5 epochs are used to train tasks with 5Hz, 10Hz, and 20Hz data, and 1 epoch is used to train the task with a larger volume of 50Hz data. The number of epochs is empirically identified by the data size of each task, since the search space of the number of epochs for each task is exponentially large.
Datasets. We evaluate our approach using permuted and split MNIST (Kirkpatrick et al., 2017), permuted and split CIFAR-10 (Ivan, 2019), split CIFAR-100 datasetsfoot_1 , and our human activity recognition in the wild (HARW) dataset. Permuted MNIST is a variant of MNIST (LeCun et al., 1998) dataset, where each task has a random permutation of the input pixels, which is applied to all the images of that task. We adopt this approach to permute the CIFAR-10 dataset, including the input pixels and three color channels. Our HARW dataset was collected from 116 users, each of whom provided mobile sensor data and labels for their activities on Android phones consecutively in three months. HARW is an ultimate task for L2M, since different sensor sampling rates, e.g., 50Hz, 20Hz, 10Hz, and 5Hz, from different mobile devices are considered as L2M tasks. The classification output includes five classes of human activities, i.e., walking, sitting, in car, cycling, and running. The data collection and processing of our HARW dataset is in Appx. D. The setting of split CIFAR-10 and CIFAR-100, and split MNIST datasets are in Appx. E.
Model Configuration. In the permuted MNIST dataset, we used three convolutional layers (32, 64, and 96 features).
In the permuted CIFAR-10 dataset, we used a Resnet-18 network (64, 64, 128, 128, and 160 features) with kernels (4, 3, 3, 3, and 3). In the HARW dataset, we used three convolutional layers (32, 64, and 96 features). Detailed model configurations are in the Appx. E. To conduct a fair comparison, we applied a grid-search for the best values of hyper-parameters, including the privacy budget ϵ ∈ [4, 10], the noise scale z ∈ [1.1, 2.5], and the clipping bound C ∈ [0.01, 1], in the NaiveGaussian mechanism. Based on the results of our hyper-parameter grid-search (Table 5), we set z = 2.2 for ϵ = 4.0, z = 1.7 for ϵ = 7.0, and z = 1.4 for ϵ = 10.0, and C = 0.01 is used for all values of ϵ.
Evaluation Metrics. We employ the well-applied average accuracy and forgetting measures after the model has been trained with all the batches up to task τ (Chaudhry et al., 2018;2019), defined as follows: (1) average accuracy τ =
where f τ t is the forgetting on task t after the model is trained with all the batches up till task τ . f τ t is computed as follows: f τ t = max l∈{1,...,τ -1} (a l,n,t -a τ,n,t ); and (3) We measure the significant difference between two average accuracy curves induced by models A and B after task τ , using a p value (2-tail t-tests) curve:
Results in Permuted MNIST. Fig. 7a and Table 1 illustrate the average accuracy and forgetting measure of each model as a function of the privacy budget ϵ on the permuted MNIST dataset. It is clear that the NaiveGaussian mechanism does not work well under a tight privacy budget ϵ ∈ [0.5, 2] given a large number of tasks m = 20. This is because each task can consume a tiny privacy budget ϵ/m resulting in either a large noise injected into the clipped gradients or a lack of training steps to achieve better model utility. By avoiding the privacy budget accumulation across tasks and training steps, our L2DP-ML models significantly outperform the NaiveGaussian mechanism. Our L2DP-ML model achieves 47.73% compared with 10.43% of the NaiveGaussian after 20 tasks given ϵ = 0.5 (p < 6.81e -15).
Regarding the upper bound performance, there is a small average accuracy gap between the noiseless A-gem model and our L2DP-ML models given a small number of tasks. The gap increases when the number of tasks increases (23.3% at ϵ = 0.5 with 20 tasks). The larger the privacy budget (i.e., ϵ = 2.0), the higher the average accuracy we can achieve, i.e., an improvement of 9.92% with p < 2.83e -14, compared with smaller privacy budgets (i.e., ϵ = 0.5). Also, our L2DP-ML models have a relatively good average forgetting with tight privacy protection (ϵ = 0.5, 1, and 2), compared with the noiseless A-gem model.
Results in Permuted CIFAR-10. Although permuted CIFAR-10 tasks are very difficult (Fig. 7b and Table 1), even with the noiseless A-gem model, i.e., 35.24% accuracy on average, the results on the permuted CIFAR-10 further strengthen our observation. Our L2DP-ML models significantly outperform the NaiveGaussian mechanism. Our L2DP-ML model achieves an improvement of 8.84% in terms of average accuracy over the NaiveGaussian after 17 tasks given ϵ = 4 (p < 4.68e -7). We further observe that the NaiveGaussian mechanism has a remarkably larger average forgetting compared with our L2DP-ML (Table 1).
Interestingly, the gap between A-gem and our L2DP-ML models is notably shrunken when the number of tasks increases (from 16.47% with 1 task to 9.89% with 17 tasks, at ϵ = 4). In addition, the average forgetting values in our L2DP-ML are better than the noiseless A-gem. This is a promising result. We also registered that the larger the privacy budget (i.e., ϵ = 10), the higher the average accuracy that we can achieve, i.e., an improvement of 4.73% with p < 1.15e -9, compared with smaller budgets (i.e., ϵ = 4).
Heterogeneous Training. We now focus on shedding light into understanding the impacts of heterogeneity and privacy on model utility given different variants of our L2DP-ML mechanisms and the noiseless A-gem model. The p value curves are in Figures 5 and6, Appx. E.
On the HARW task (Fig. 3c and Table 1), our L2DP-ML model achieves a very competitive average accuracy, given a very tight DP budget ϵ = 0.2 (i.e., 61.26%) compared with the noiseless A-gem model (i.e., 62.27%), across four tasks. Our model also achieves a better average forgetting, i.e., 11.33, compared with 12.69 of the noiseless A-gem model. That is promising. Increasing the privacy budget modestly increases the model performance. The differences in terms of average accuracy and forgetting are not significant. This is also true, when we randomly flip the order of the tasks (Fig. 4 and Table 2). The results showed that our model effectively preserves Lifelong DP in HARW tasks.
Heterogeneous training, with customized numbers of epochs and task orders, further improves our model performance, under the same Lifelong DP protection. Fig. 5 illustrates the p values between the average accuracy curves of our L2DP-ML, given 1) heterogeneous training with different numbers of epochs, 2) task orders, and 3) privacy budgets, over its basic settings, i.e., ϵ = 0.5 for the permuted MNIST dataset, ϵ = 4 for the permuted CIFAR-10 dataset, and ϵ = 0.2 for the HARW dataset, with one training epoch.
• In the permuted MNIST dataset (Figs. 7a and5a), when our L2DP-ML model is trained with 2 or 3 epochs per task, the average accuracy is improved, i.e., 2.81%, 4.8% given 2, 3 epochs, respectively, with p < 8.44e -9. In the permuted CIFAR-10, using larger numbers of training epochs shows significant performance improvements over a small number of tasks (Fig. 5b). When the number of tasks becomes larger, the p values become less significant (even insignificant), compared with the p value curves of larger DP budgets (i.e., ϵ = 2 and ϵ = 10 in the permuted MNIST and permuted CIFAR-10). Meanwhile, training with a larger number of epochs yields better results with small numbers of tasks (i.e., fewer than 6 tasks), compared with larger DP budgets.
• In the HARW tasks, the improvement is more significant (Figs. 3c and5c). Heterogeneous and Balanced L2DP-ML models outperform the basic settings with uniform numbers of training epochs, i.e., 1, 2, and 5 epochs. On average, we registered an improvement of 1.93% given the Balanced L2DP-ML and an improvement of 5.14% given the Heterogeneous L2DP-ML, over the basic setting (1 training epoch). The results are statistically significant (Fig. 5c). The average forgetting values of the Balanced L2DP-ML (0.1593) and the Heterogeneous L2DP-ML (0.1920) are higher than the basic setting (0.1133), with p < 2.19e -5 (Table 1). This is expected as a primary trade-off in L2M, given a better average accuracy. In fact, the average forgetting values are also notably higher given larger uniform numbers of epochs, i.e, 2 and 5 epochs, and the Balanced A-gem. We do not address this fundamental issue in L2M since it is out-of-scope of this study. We focus on preserving Lifelong DP.
• We observe similar results in randomly flipping the order of the tasks (Figs. 4 and6, Table 2). Among all task orders, our Heterogeneous L2DP-ML achieves the best average accuracy (66.4%) with the task order [5Hz, 10Hz, 20Hz, 50Hz] (Fig. 3c) compared with the worse order [20Hz, 50Hz, 5Hz, 10Hz] (56.69%) (Fig. 4b), i.e., p < 9.9e -5. More importantly, in both average accuracy and forgetting, our Balanced and Heterogeneous L2DP-ML models achieve a competitive performance compared with the noiseless Balanced A-gem, which is considered to have the upper bound performance, and a better performance compared with having the uniform numbers of epochs across tasks. This obviously shown that the distinct ability to offer the heterogeneity in training across tasks greatly improves our model performance, under the same Lifelong DP protection.
Results in Split Tasks. We observe similar results on split CIFAR-10, CIFAR-100, and MNIST datasets as L2DP-ML achieves competitive average accuracy approaching the noiseless A-gem model under rigorous privacy budgets (Fig. 7, Appx. E). After 5 tasks of the split MNIST dataset, L2DP-ML achieves 73.54% and 81.83% in average accuracy at the privacy budgets 0.5 and 1 respectively, compared with 79.71% of the noiseless A-gem. Interestingly, our L2DP-ML has slightly higher average accuracy than the noiseless A-gem after 11 tasks of the split CIFAR-10 and CIFAR-100 dataset (14.83% in L2DP-ML at ϵ = 4 compared with 13.44% in the noiseless A-gem). One reason is that Lifelong DP-preserving noise can help to mitigate the catastrophic forgetting. As showed in Table 6 (Appx. E), our L2DP-ML obtains a significantly lower average forgetting (2.7% at ϵ = 4) than the noiseless A-gem (19.5%).
In this paper, we showed that L2M introduces unknown privacy risk and challenges in preserving DP. To address this, we established a connection between DP preservation and L2M, through a new definition of Lifelong DP. To preserve Lifelong DP, we proposed the first scalable and heterogeneous mechanism, called L2DP-ML. Our model shows promising results in several tasks with different settings and opens a long-term avenue to achieve better model utility with lower computational cost, under Lifelong DP.
Proof 1 Let us denote A 1:i as A 1 , . . . , A i , we have:
Consequently, Theorem 1 does hold.
Proof 2 ∀τ ∈ T, let D τ and D
Therefore, we further have
In addition, we also have that:
where D = {D 1 , . . . , D m }.
Together with Eq. 17, by having disjoint and fixed datasets in the episodic memory, we have that:
Without loss of the generality, we can assume that x e ∈ D τ : Eqs. 16 -18
This is also true when x e ∈ D ref and x e ̸ ∈ D τ .
As a result, we have
After one training step, D τ will be placed into the episodic memory M τ to create the memory M τ +1 . In the next training task, D τ can be randomly selected to compute the episodic gradient g ref . This computation does not incur any additional privacy budget consumption for the dataset D τ , by applying the Theorem 4 in (Phan et al., 2020), which allows us to compute gradients across an unlimited number of training steps using R Dτ (θ τ -1
) and L Dτ (θ τ -1
). Therefore, if the same privacy budget is used for all the training tasks in T, we will have only one privacy loss for every tuple in all the tasks. The optimization in one task does not affect the DP guarantee of any other tasks. Consequently, we have
Eq. 21 can be further used to prove the Lifelong DP protection. Given data m where M t = D t in Alg. 1, we have that
Therefore, we have
where
1. Since all the datasets are non-overlapping, i.e., ∩ i∈[1,m] D i = ∅, given an arbitrary tuple x e , we have that
Thus, the optimization of {θ i 1 , θ i 2 } = arg min θ1,θ2 [R Di (θ i-1 1 ) + L Di (θ i-1
2 )] for any other task i different from τ does not affect the privacy protection of x e in D. From Eqs. 23 and 24, we have
The worse privacy leakage case to x e is that D τ is used in every D
In order to bound the privacy loss, we consider the worse case; therefore, from Eq. 25, we further have that
Eq. 26 is equivalent to the continuously training of our model by optimizing R and L with D τ used as both the current task and the episodic memory, across m steps. By following the Theorem 4 in (Phan et al., 2020), the privacy budget is not accumulated across training steps. Therefore, we have that
In the least privacy leakage case, we have that
As a result, we have that
where ϵ = (ϵ 1 + ϵ 1 /γ x + ϵ 1 /γ + ϵ 2 ).
From Eqs. 27 and 29, we have that Alg. 1 achieves
. Consequently, Theorem 2 does hold.
Algorithm 2 L2DP-ML with Streaming Batch Training Input:
, batch size λ, privacy budgets: ϵ 1 and ϵ 2 , learning rate ϱ Output: Compute Gradients:
)} with the noise χ3 λ 14:
Randomly Select a batch B ∈ B 18:
Data Collection. We utilize Android smartphones to collect smartphone sensor data "in the wild" from university students as subjects for the following reasons: (1) University students should have relatively good access to the smartphones and related technologies;
(2) University students should be more credible and easier to be motivated than other sources (e.g., recruiting test subjects on crowd-sourcing websites); and (3) It will be easier for our team to recruit and distribute rewards to students. We launched two data collection runs at two universities for three months each. During the course of three months, we let the participants to collect data and labels by themselves (in the wild), and only intervene through reminding emails if we saw a decline in the amount of daily activities. A total of 116 participants were recorded after the two data collection runs.
Data Processing. For the demonstration purpose of this paper, we use only accelerometer data. Our data processing consists of the following steps: (1) Any duplicated data points (e.g., data points that have the same timestamp) are merged by taking the average of their sensor values;
(2) Using 300 milliseconds as the threshold, continuous data sessions are identified and separated by breaking up the data sequences at any gap that is larger than the threshold; (3) Data sessions that have unstable or unsuitable sampling rates are filtered out. We only keep the data sessions that have a stable sampling rate of 5Hz, 10Hz, 20Hz, or 50Hz; (4) The label sessions that are associated with each data session (if any) are identified from the raw labels. Note that the label sessions are also filtered with the following two criteria to ensure good quality: (a) The first 10 seconds and the last 10 seconds of each label session are trimmed, due to the fact that users were likely operating the phone during these time periods; (b) Any label session longer than 30 minutes is trimmed down to 30 minutes, in order to mitigate the potential inaccurate labels due to users' negligence (forgot to turn off labeling); and (5) We sample data segments at the size of 100 data points with sliding windows. Different overlapping percentages were used for different classes and different sampling rates. The majority classes have 25% overlapping to reduce the number of data segments, while the minority classes have up to 90% overlapping to increase the available data segments. The same principle is applied to sessions with different sampling rates. We sample 15% of data for testing, while the rest are used for training (Table 3). Data Normalization. In our L2DP-ML models, we normalize the accelerometer data with the following steps: (1) We compute the mean and variance of each axis (i.e., X, Y , and Z) using only training data to avoid information leakage from the training phase to the testing phase. Then, both training and testing data are normalized with zscore, based on the mean and variance computed from training data; (2) Based on this, we clip the values in between [min, max] = [-2, 2] for each axis, which covers at least 90% of possible data values; and (3) Finally, all values are linearly scaled to [-1, 1] to finish the normalization process, as x = 2 × [ x-min max-min -1/2]. In the HARW dataset, each data tuple includes 100 values × 3 channels of the accelerometer sensor, i.e., 300 values in total as a model input. The classification output includes five classes of human activities, i.e., walking, sitting, in car, cycling, and running (Table 3, Appx. D). Given 20Hz, 5Hz, 10Hz, and 50Hz tasks, we correspondingly have 881, 7553, 621, and 156,033 data points in training and 159, 1,297, 124, and 27,134 data points in testing.
Baseline Model Performance. We conducted experiments on the HARW dataset in a centralized training on the whole dataset including all the data sampling rates using following baselines: 1) CNN-based model with the numbers of convolution-channels set to 32, 64, 128, denoted as CNN-32, CNN-64, CNN-128, respectively; 2) Bidirectional LSTM (BiLSTM); and 3) CNN-based models proposed by Ignatov (2018), with additional features (CNN-Ig) and without additional features (CNN-Ig-featureless) using the Ignatov's recommended settings in Ignatov (2018).
As in Table 4, our model trained on each task independently achieves competitive results with these baselines under a rigorous DP budget (ϵ = 0.2), i.e., 77%, 76%, 75%, 58%, on the 5Hz, 10Hz, 20Hz, and 50Hz learning tasks respectively. Although the number of 50Hz training data points is larger than other tasks, the data labels are noisy and collected in short-time periods due to the limited computational resources on mobile devices; thus, the model performance in the 50Hz learning task is lower.
Model Configuration. In the permuted MNIST and the Split MNIST datasets, we used three convolutional layers (32, 64, and 96 features). Each hidden neuron connects with a 5x5 unit patch. A fully-connected layer has 512 units. In the permuted CIFAR-10 and the Split CIFAR-10/100 datasets, we used a 64,128,128,and 160 features) with kernels (4, 3, 3, 3, and 3). One fully-connected layer has 256 neurons. In the HARW dataset, we used three convolutional layers (32, 64, and 96 features). Each hidden neuron connects with a 2x2 unit patch. A fully-connected layer has 128 units.
In the Split CIFAR-10 and CIFAR-100 setting, there are 11 tasks, in which the first task is the full CIFAR-10 classification task, and the remaining 10 tasks consist of splits from the CIFAR-100 dataset. Each split contains 10 classes from the CIFAR-100. We adopt this approach from (Von Oswald et al., 2020). In the Split MNIST setting, there are 5 tasks, in which each task consists of 2 classes from the MNIST dataset. There is no overlapping classes between tasks in the Split CIFAR-10 and CIFAR-100, and in the Split MNIST.
In order to be fair in comparison with the L2DP-ML and A-gem mechanisms, we conducted experiments over a wide range of privacy hyper-parameters such as privacy budget (ϵ), noise scale (z), and sensitivity to select the best hyper-parameters in NaiveGaussian mechanism in our experiments. The search ranges and their results (i.e., average accuracy over all tasks) are provided in Table 5. We reported the best results, i.e., highest average accuracy over all tasks, of the hyper-parameter grid-search experiments.
https://en.wikipedia.org/wiki/Operator_norm
Datasets were downloaded and evaluated by Phung Lai, Han Hu, and NhatHai Phan.