Despite remarkable performance in many applications, natural language models (NLMs), such as GPT models [1,2,3], are vulnerable to privacy attacks because of such attacks' capacity to memorize unique patterns in training data [4]. Recent data training extraction attacks [5] illustrate that sensitive entities, such as a person's name, email address, phone number, physical address, etc., can be accurately extracted from NLM parameters. These sensitive entities and the language data memorized in NLMs may identify a data owner -explicitly by name or implicitly, e.g., via a rare or unique phrase -and link that data owner to extracted sensitive entities.
Our main goal is to provide a rigorous guarantee that a trained NLM protects the privacy of sensitive entities in the training data and the participation information (membership) of the data owners in learning the model while maintaining high model utility. The simple solution of anonymizing (including removing/de-identifying) sensitive entities is insufficient; since the anonymized entities can be matched with non-anonymized data records in another dataset [6]. Also, the model utility can be notably affected, as shown in our experimental study. While cryptographic approaches can be applied to protect privacy, they introduce computation and resource overhead [7]. Therefore, we proposed to use differential privacy [8], one of the adequate solutions, given its formal protection without undue sacrifice in computation efficiency and model utility.
Differential privacy (DP) provides rigorous privacy protection as a probabilistic term, limiting the knowledge about a data record an ML model can leak while learning features of the whole training set. DP-preserving mechanisms have been investigated and applied in real-world [9,10,11], including image processing [12], healthcare data [13], financial records [14], social media [15], and NLMs [16,17,18,19].
However, existing DP protection levels, including samplelevel DP [6,9,20,21], user-level DP [16,22], element-level DP [23], and local (feature-level) DP [17,18,24,25], do not provide the privacy protection level demanded to solve our problem. Given training data: 1) Sample-level DP protects the privacy of a single sample; 2) User-level DP protects privacy of a single data owner, also called a single user, who may contribute one or more data samples; 3) Element-level DP partitions data owners' contribution to the training data into sensitive elements, e.g., a curse word, which will be protected. Element-level DP does not provide privacy protection to data owners; and 4) Local (feature-level) DP protects true values of a data sample from being inferred. Recently, [18] proposed local DP-preserving approaches for text embedding extraction under (word-level) local DP (Eq. 2). However, the privacy budget in [18] is accumulated over the dimensions of embedding, resulting in an impractical (loose) privacy guarantee (Appendix D in our supplemental document 1 ).
Therefore, there is a demand for a new level of DP to protect privacy simultaneously for both sensitive entities in the training data and the participation information of data owners in learning NLMs. Motivated by this, we structure our paper around the following significant contributions.
• We propose a novel notion of user-entity adjacent databases (Definition 2), leading to formal guarantees of userentity privacy rather than privacy for a single user or a single sensitive entity.
• To preserve UeDP, we introduce a novel algorithm, called UeDP-Alg, which leverages the recipe of DP-FEDAVG [16] to protect both sensitive entities and user membership under DP via the moments accountant [9]. Moments accountant was first developed to preserve DP in stochastic gradient descent (SGD) for sample-level privacy. Our federated averaging approach groups multiple SGD updates computed from a two-level random sampling process, including a random sample of users and a random sample of sensitive entities. That enables large-step model updates and optimizes the trade-off between privacy loss and the model utility through a tight noise scale bound (Lemma 1 and Theorem 1).
• Through theoretical analysis and rigorous experiments conducted on benchmark datasets, we show that our UeDP-Alg outperforms baseline approaches in terms of model utility on fundamental tasks, i.e., next word prediction and text classification, under the same privacy budget consumption. Our code is available 2 .
In this section, we revisit NLM tasks, privacy risk, and DP. For the sake of clarity, let us focus on the next word prediction, and we will extend it to text classification in Section VI. A list of sensitive entity categories is summarized in Table I.
a) Next Word Prediction: Let D be a private training data containing U users (data owners) and a set of sensitive entities E. Each user u ∈ U consists of n u sentences. Given a vocabulary V, each sentence is a sequence of words, presented as x = x 1 x 2 . . . x mu , where
) is a word in x and m u is the length of x. In next word prediction, the first j words in x, i.e., x 1 , x 2 , . . . , x j (∀j < m u ), are used to predict the next word x j+1 . Here, x j+1 can be considered as a label in the next word prediction task. Perplexity P P = 2 -x∈D p(x) log 2 p(x) is a measurement of how well a model predicts a sentence and is often used to evaluate language models, where p(x) is a probability to predict the next word x j+1 in x [26]. A lower perplexity indicates a better model. b) Sensitive Entities and Sentences: Each sensitive entity e ∈ E consists of a word or consecutive words that must be protected. For instance, personal identifiable information (PII) related to an identifiable person, such as person names, locations, and phone numbers, can be considered sensitive entities. If a sentence x consists of a sensitive entity e, x is considered as a sensitive sentence; otherwise, x is a nonsensitive sentence.
For instance, in Fig. 1, "David Johnson," "Maine," "September 18," and "Main Hospital" are considered sensitive entities, correspondingly categorized into PII, geopolitical entities (GPE) (i.e., countries, cities, and states), time, and organization names. The first and second sentences consisting of the sensitive entities are considered sensitive sentences. Meanwhile, the third and fourth sentences are non-sensitive since they do not contain any sensitive entities. c) Privacy Threat Models: It is well-known that trained ML model parameters can disclose information about training data [5,27], especially in NLMs [5,16]. Given a data sample and model parameters, by using a membership inference attack [28,29,30], adversaries can infer whether the training used the sample or not. In NLMs, adversaries can accurately recover individual training examples, such as full names, email addresses, and phone numbers of individuals, using training data extracting attacks [5]. Accessing to these can lead to severe privacy breaches. with a privacy budget ϵ and a broken probability δ.
The privacy budget ϵ controls the amount by which the distributions induced by D and D ′ may differ. A smaller ϵ enforces a stronger privacy guarantee. The broken probability δ means the highly unlikely "bad" events, in which an adversary can infer whether a particular data sample belongs to the training data, happen with the probability ≤ δ.
There are different levels of DP protection in literature categorized into four research lines, including sample-level DP, user-level DP, element-level DP, and local (feature-level) DP. They are different from our goal since we focus on providing simultaneous protections to data owners and sensitive entities in textual data. Let us revisit these DP levels and distinguish them with our goal.
a) Sample-level DP: Traditional DP mechanisms [6,20,31] ensure DP at the sample-level, in which adjacent datasets D and D ′ are different from at most a single training sample. Sample-level DP does not protect privacy for users. That is different from our goal. We aim at protecting privacy for users and sensitive entities, which are different from data samples.
b) User-level DP: To protect privacy for users, who may contribute more than one training sample, rather than a single sample, [16] proposed a user-level DP, in which neighboring databases D and D ′ are defined to be different from all of the samples associated with an arbitrary user in the training set. Several works follow this direction [22,32]. User-level DP differs from our goal, since it does not provide privacy protection for sensitive entities in the training set.
c) Element-level DP: [23] introduce element-level DP, in which users are partitioned based on sensitive elements, which are protected in a way that an adversary cannot infer whether a user has a sensitive element in her/his data, e.g., if a user has ever sent a curse word in his/her messages or not. Similar to sample-level DP, element-level DP is different from our goal, since it does not provide DP protection for users.
d) Local (feature-level) DP: [18] proposed a notion of word-level local DP for a sentence's embedding features, in which two adjacent sentences x and x ′ are different at most one word:
where f (x) extracts embedded features of x and A is a randomized algorithm, such as a Laplace mechanism [6]. In a similar effort, [17] applied a randomized response mechanism [24,33,34] on top of binary encoding of embedded features' real values to achieve local DP feature embedding. The approaches proposed in [17,18] are different from our goal, since they do not offer either user-level DP or word-level DP.
In this section, we focus on answering the question: "Could we protect sensitive entities and user membership simultaneously by leveraging existing levels of DP and how?" Based upon that, we propose our user-entity DP notion.
To protect sensitive entities and user membership, a potential approach is to decouple them into separated protection levels offering by existing DP notions. However, this approach has limitations as discussed next.
Let us consider a sentence consisting of one or more than one sensitive entities. We can leverage sample-level DP to protect the sentence, i.e., each sentence could be a sample, covering all the sensitive entities under DP. If each user has only one sentence, then this approach can also protect the user membership. In practice, one user may contribute many sentences to the training data. To address this issue, we can utilize group privacy [6] resulting in an amplification of the privacy budget proportional to the number of sentences a user may have in the training data.
Instead of group privacy, another potential solution is applying user-level DP on top of the sample-level DP to protect both sentence and user membership. In the samplelevel DP, we can clip and inject Gaussian noise into the gradient derived from each sentence [9]. Meanwhile, in the user-level DP, an additional Gaussian noise is injected into the aggregation of gradients, each of which derived from a single user [16]. Although this combination of sample -user levels can cover both sensitive entities and user membership under DP protection, it has disadvantages. First, some sentences are sensitive and other sentences are not. Protecting all (sensitive and non-sensitive) sentences or removing all the sensitive sentences from the training data may cause significant model utility degradation. Second, different sentences may consist of different types and numbers of sensitive entities. Under the same sampling probability for training as in [9] for samplelevel DP, these sentences expose different privacy risks to user identity and sensitive entities.
To address these issues, instead of the sentence level, one can work at the word level by extracting embedded features for every words in the training data. Embedded features of sensitive entities are randomized by local DP-preserving mechanisms [34]. The randomized embedded features are aggregated with embedded features of non-sensitive words to train NLMs. Then, user-level DP can be applied to clip gradients derived from each user's data with adding Gaussian noise into the aggregation of these gradients. However, this approach suffers from a remarkable model utility degradation.
Local DP provides rigorous privacy protection but it comes with a cost in terms of utility [35]. Then, adding the userlevel DP adversely affects the utility.
The root cause of these limitations is that the combination of sentence-level DP and user-level DP notions does not capture the correlation between sensitive entities and user membership in unifying notion of DP. Meanwhile, working with wordlevel embedded features under local DP introduces expensive model utility costs. Therefore, there is a demand for a unifying notion of DP and an optimal approach to protect both sensitive entities and user membership in training NLMs.
To preserve privacy for both users and sensitive entities in NLMs, we propose a new definition of user-entity adjacent databases, as follows: Two databases D and D ′ are user-entity adjacent if they differ in a single user and a single sensitive entity; that is, one user u ′ and one sensitive entity e ′ are present in one database (i.e., D ′ ) and are absent in the other (i.e., D). Together with the absence of all sentences from the user u ′ in D, all sentences (across users) consisting of the sensitive entity e ′ are also absent in D. This is because one user can have multiple sentences, and one sensitive entity can exist in multiple sentences for training. The definition of our userentity adjacent databases is presented as follows: Given the user-entity adjacent databases, we present our UeDP in the following definition. Definition 3. (ϵ, δ)-UeDP. A randomized algorithm A is (ϵ, δ)-UeDP if for all outcomes O ⊆ Range(A) and for all user-entity adjacent databases D and D ′ , we have:
with a privacy budget ϵ and a broken probability δ.
UeDP provides rigorous privacy protection to both users and sensitive entities; however, the practicability of UeDP preservation depends on the reliability of sensitive entity detection from the training text data. In practice, misidentifying sensitive entities can introduce extra privacy risks. In addition to addressing this challenge, we focus on bounding the sensitivity of an NLM under UeDP and addressing the trade-off between privacy loss and model utility.
Identifying all the sensitive entities typically requires intensive manual efforts [36]. We are aware of this issue in real-world applications. Fortunately, there are several ways to automatically identify sensitive entities in textual data, such as: 1) Using Named Entity Recognition (NER) [37,38]; and 2) Using publicly available toolkits for detecting named entities or PII in text, e.g., spaCy [39], Stanza [40], and Microsoft Presidio 3 . These approaches and toolkits are user-friendly and reliable to reduce manual efforts in identifying sensitive entities and information. We found that the results from spaCy cover over 94% of sensitive information identified by Amazon Mechanical Turk (AMT) workers in a diverse set of datasets used in our experiments. More information about identifying sensitive entities is available in Appx. A. 3 https://microsoft.github.io/presidio/ Although effective, the small error rate (i.e., ≊ 6%) from these techniques introduces a certain level of privacy risk, that means, some sensitive entities may be misidentified to be nonsensitive, and vice-versa. Classifying non-sensitive entities to be sensitive entities does not incur any extra privacy risk. Meanwhile, classifying one (or more than one) sensitive entity to be non-sensitive in a sentence introduces two issues, as follows: (1) There may be sensitive sentences misidentified to become non-sensitive sentences. In order words, given a set of non-sensitive sentences detected by NER tools, we do not know which sentence is truly non-sensitive; and (2) Given a sensitive sentence x, some sensitive entities in x may not be identified by NER tools. Preserving UeDP in NLMs by directly using the results of NER tools will expose these misidentifying sensitive sentences and entities unprotected.
To address the problem of sensitive entity misidentification in preserving UeDP, our key idea is:
(1) Extending UeDP by considering each sentence, identified to be non-sensitive using NER tools [39,40], in the private training dataset as a single type of sensitive entity. We denote this extended set of sensitive entities as S. The private dataset D now consists of U users and a (sufficient) set of sensitive entities E ∪ S that will be protected.
(2) Upon forming the sufficient set of sensitive entities, we propose a two-step sampling approach to strictly preserve UeDP in NLMs. In our approach, at a training round t, we sample a set of users from U and a set of sensitive entities from E ∪ S. We use sentences in the training data of the sampled users consisting of the sampled sensitive entities to train NLMs. In this sampling approach: (i) If a sensitive sentence x is not sampled for training, i.e., due to the fact that some sensitive entities in x are not identified by NER tools, x is not used for training at the round t; thus avoiding privacy risks exposed by x; and (ii) If the sensitive sentence x is sampled for training, then the sensitive entities in x, which are not identified by NER tools, are protected since x is protected under DP.
By covering all possible cases of sensitive entity misidentification, we strictly preserve UeDP without having additional privacy risks. The pseudo-code of our algorithm is in Alg. 1.
At each iteration t, we randomly sample U t users from U , E t detected sensitive entities from E, and S t extended sensitive entities from S, with sampling rates q u , q e , and q s , respectively (Lines 8 and 10). Then, we use all sensitive sentences in E t u ∪ S t u consisting of the sensitive entities in E t and S t belonging to the selected users in U t for training. Like [16], we leverage the basic federated learning setting in [41] to compute gradients of model parameters for a particular user, denoted as ∆ t+1
u,E (Line 11). Here, we clip the peruser gradients so that its l 2 -norm is bounded by a predefined gradient clipping bound β (Lines 20 -29). Next, a weightedaverage estimator f E + is employed to compute the average gradient ∆ t+1 using the clipped gradients ∆ t+1 u,E gathered from all the selected users (Line 13). Finally, we add random Gaussian noise N (0, Iσ 2 ) to the model update (Line 15). During the training, the moments accountant M is used to compute the T training steps' privacy budget consumption (Lines 16 -18).
To tighten the sensitivity bound, our weighted-average estimator f E + (Line 13) is as follows:
where ∆ t+1 u,E = e∈E t u w e ∆ u,e + s∈S t u w s ∆ u,s , and w u , w e , and w s ∈ [0, 1] are weights associated with a user u, a detected sensitive entity e, and an extended sensitive entity s.
These weights capture the influence of a user and sensitive entities to the model outcome. ∆ u,e and ∆ u,s are the parameter gradients computed using the sensitive entities e ∈ E and s ∈ S. In addition, W u = u∈U w u , W e = e∈E w e , and W s = s∈S w s .
Since E[ e∈E t u w e + s∈S t u
w s ] = q e W e + q s W s , the estimator f E + is unbiased. The sensitivity of the estimator
u w e ( s consists of e ∆ u,s ) q u W u (q e W e + q s W s ) + u∈U t ∪u ′ w u s∈S t u w s ∆ u,s q u W u (q e W e + q s W s ) + u∈U t ∪u ′ w u w e ′ ( s consists of e ′ ∆ u,s ) q u W u (q e W e + q s W s )
u w e ( s consists of e ∆ u,s ) q u W u (q e W e + q s W s )
Consequently, Lemma 1 holds.
Once the sensitivity of the estimator f E + is bounded, we can add Gaussian noise scaled to the sensitivity S(f E + ) to obtain a privacy guarantee. By applying Lemma 1, the noise scale σ becomes:
The noise scale σ in Eq. 6 is tighter than the noise scale in existing works [16,22] proportional to the number of sensitive entities used in the training process (i.e., q e W e + q s W s ). return ClipFn(θ -θ 0 , β)
Therefore, we can inject less noise into our model under the same privacy budget while improving our model utility.
In extreme cases, that is also true: (1) E is empty, which means there are no detected sensitive entities. Given a fixed set of training data, while E is empty, S becomes larger (i.e., covering the whole dataset), resulting in a larger value of W s . Therefore, we obtain a larger value of q s W s (with a predefined q s ), enabling us to reduce the noise scale under the same UeDP guarantee. That is an advantage compared with the naive approach that only uses detected sensitive entities E in the training process (i.e., ignoring the term q s W s in Eq. 6). If E is empty, the naive approach will have no sentences for training; and (2) S is empty, that is, every sentence in the data consists of at least one detected sensitive entity e ∈ E. Similarly, given a fixed set of training data, if S is empty, then E and W e become larger. It enables us to obtain a larger value of q e W e (with a pre-defined q e ), which results in smaller noise scale while maintaining the high model utility.
UeDP Guarantee. Given the bounded sensitivity of the estimator, the moments accountant M [9] is used to get a tight bound on the total UeDP privacy consumption of T steps of the Gaussian mechanism with the noise N (0, Iσ 2 ) (Line 15). Theorem 1. For the estimator f E + , the moments accountant of the sampled Gaussian mechanism correctly computes UeDP privacy loss with the scale z = σ/S(f E + ) for T training steps.
Proof. At each step, users, detected sensitive entities in E, and extended sensitive entities in S are selected randomly with probabilities q u , q e , and q s , respectively. For f E + , if the l 2 -norm of each user's gradient update, using the sampled sensitive entities in E t u ∪ S t u , is bounded by S(f E + ), then the moments accountant can be bounded by that of the sampled Gaussian mechanism with sensitivity 1, the scale z = σ/S(f E + ), and sampling probabilities q u , q e , and q s . Thus, we can apply the composability as in Theorem 2.1 [9] to correctly compute the UeDP privacy loss with the scale z = σ/S(f E + ) for T steps.
We conducted an extensive experiment, both in theory and on benchmark datasets, to shed light on understanding 1) the integrity of sensitive entity identification, 2) the interplay among the UeDP privacy budget (ϵ, δ), different types of sensitive entities (i.e., organization, location, PII, and miscellaneous entities), and model utility, and 3) whether considering the extended set of sensitive entities S will improve our model utility under the same UeDP protection.
a) Baseline Approaches: We evaluate our UeDP-Alg in comparison with both noiseless and privacy-preserving mechanisms (either user level or entity level), including: (1) User-level DP [16], which is the state-of-the-art DP-preserving model closely related to our work; (2) De-Identification [42], which is considered as a strong baseline to protect privacy for sensitive entities. Although sensitive entities are masked to hide them in the training process, De-Identification does not offer formal privacy protection to either the data owners or sensitive entities; and (3) A Noiseless model, which is a language model trained without any privacy-preserving mechanisms. In addition, we consider the naive approach, which is a variation of our algorithm, called UeDP-Alg f E . As a baseline, the estimator f E is computed without taking the extended set of sensitive entities S into account (Appx. B, Supplementaryfoot_1 ). This is further used to comprehensively evaluate our proposed approach. In our experiment, our algorithms and baselines, i.e., UeDP-Alg, User-level DP, and De-Identification, are applied on the noiseless model in the training process. As in the literature review [18,23], there are no other appropriate DPpreserving baselines for UeDP protection.
b) Evaluation Tasks and Metrics: Our experiment considers two tasks: (1) next word prediction and (2) text classification. For the next word prediction, we employ the widely used perplexity [43]. The smaller perplexity is, the better model is. For the text classification, we use the test error rate as in earlier work [44]. Test error rate implies prediction error on a test set, so it is 1 -the test set's accuracy. The lower the test error rate is, the better model is.
c) Data and Model Configuration: For the reproducibility sake, all details about our datasets and data processing are included in Appx. C (Supplementary 4 ). We carried out our experiment on three textual datasets, including the CONLL-2003 news dataset [37], AG's corpus of news articles 4 , and our collected Security and Exchange Commision (SEC) financial contract dataset. The data breakdown is in Table II.
For the next word prediction, we employ a GPT-2 model [2], which is one of the state-of-art text generation models. To make the work easily reproducible, we use a version of the pretrained GPT-2 that has 12-layer, 768-hidden, 12heads, 117M parameters, and then fine-tune with the aforementioned datasets as our Noiseless GPT-2 model. For the text classification, we fine-tune a Noiseless BERT (i.e., BERT-Base-Uncasedfoot_2 ) pre-trained model [45] that has 12-layer, 768hidden, 12-heads, and 110M parameters with an additional softmax function on top of the BERT model. Adam optimizer is used with the learning rate is 10 -5 . Gradient clipping bound β = 0.1 and the scale z = 2.5. The sampling rates for users, detected sensitive entities, and extended sensitive entities q u , q e , and q s are 0.05, 0.5, and 1.0.
To test the effectiveness and adaptability of our mechanism across models, we also conducted experiments with an AWD-LSTM model [46], which has a much fewer parameters compared with GPT-2 and BERT. In AWD-LSTM model, we use a three-layer LSTM model with 1, 150 units in the hidden layer and an embedding input layer of size 100. Embedding weights are uniformly initialized in the interval [-0.1, 0.1] with dimension d = 100 and other weights are initialized between [-
where H is the size of all hidden layers. The values used for dropout on the embedding layer, the LSTM hidden-to-hidden matrix, and the final LSTM layer's output are 0.1, 0.3, and 0.5, respectively. Gradient clipping bound β = 0.1 and the scale z = 2. The sampling rates q u , q e , and q s are 0.05, 0.5, and 1.0 (note that q s is 0.6 in the text classification task). SGD optimizer is used. In the text classification with the AG dataset, a softmax layer is applied on top of the AWD-LSTM with the output dimension is 4, corresponding to four classes in the AG dataset. The same sets of sensitive entity categories are used for all models in the next word prediction and the text classification tasks.
Results: To answer our evaluation questions, we conducted the following experiments: (1) examining how the sensitive entities detected by the entity recognition spaCy [39] covers the sensitive information clarified by AMT workers, (2) comparing estimators f E , f E + , and User-level DP; (3) investigating the interplay between privacy budget and model utility; (4) studying the impacts of different sensitive Our result is as follows:
• Integrity of sensitive entities. Our work utilizes spaCy [39], one of the state-of-the-art large-scale entity recognition systems, to identify sensitive entities for evaluation purposes on datasets that do not have ground-truth sensitive entities, including the AG and SEC datasets. For CONLL-2003, we consider the labels of four sensitive entity types (i.e., location, person, organization, and miscellaneous) from NER as the ground truth. To evaluate the integrity of identified sensitive entities, we conducted a clarification on AMT. We found that the results from spaCy cover over 94% of sensitive information as identified by AMT workers. We recruited master-level AMT workers for a high quality of results, and we provided detailed guidance before AMT workers conducted the task. Each sentence was assigned to 3 workers to mitigate bias and subjective views. Consequently, our experiments using the spaCy identified sensitive entities are solid.
• Comparing Estimators f E , f E + , and User-level DP. In this analysis, we set q u = 0.05, q e = 0.5, q s = 1, z = 2, and compute privacy budget ϵ at δ = 10 -5 (a typical value of δ in DP) as a function of the training steps T . Fig. 4 shows curves of using different estimators and the User-level DP with all entities in CONLL-2003, AG, and SEC datasets.
Our UeDP-Alg with f E + achieves a notably tighter privacy budget compared with f E and the User-level DP in all scenarios in CONLL-2003, AG, and SEC datasets. The key reason is that typically detected sensitive entities in E appear rarely in a dataset compared with extended sensitive entities. Thus, using only sensitive entities in E identified by the spaCy in training will cause information distortion, which can damage model utility and a loose privacy budget.
User-level DP consumes a much higher privacy budget ϵ compared with both of our estimators f E + and f E . For Significantly, the privacy budget (ϵ) gap between User-level DP, f E , and f E + is proportionally increased to the number of steps T . That means the more training steps T , the larger ϵ our model can save compared with User-level DP. That is a promising result in the context that our model provides DP protection for both users and sensitive entities, compared with only protection for users in User-level DP. We observe a similar phenomenon on different sensitive categories.
• Privacy Budget (ϵ, δ)-UeDP and Model Utility. From our theoretical analysis, f E + is better than the estimator f E . Therefore, for the sake of simplicity, we only consider UeDP-Alg f E + instead of showing results from both estimators. From now, UeDP-Alg is used to indicate the use of our estimator f E + . Fig. 2 illustrates the perplexity as a function of the privacy budget ϵ for an GPT-2 model trained on a variety of sensitive entity categories in UeDP, User-level DP, and De-Identification. The noiseless GPT-2 (for the next word prediction) and BERT (for the text classification) models are considered an upper-bound performance mechanism without offering any privacy protection.
In the CONLL-2003 dataset (Fig. 2a), there are NER labels for person, location, organization, and miscellaneous entities; therefore, we choose these types as sensitive entity categories to protect in UeDP-Alg. UeDP-Alg achieves a better perplexity compared with User-level DP under a tight privacy budget ϵ ∈ Results on AG and SEC datasets (Figs. 2b and2c) further strengthen our observations. In AG and SEC datasets, we applied spaCy to identify different sensitive entity categories, such as GPE, location, organization, and PII (i.e., person and location information). UeDP-Alg achieves better results compared with User-level DP in all considering sensitive entity categories and privacy budgets, and outperforms De-Identification in most cases. That is promising and consistent with our previous analysis. For instance, in the AG dataset, at ϵ = 0. • Sensitive Entity Categories. In all datasets (Figs. 2 and9, Appx. E, Supplementary 4 ), the more sensitive sentences to protect, the higher the privacy budget is needed, and the lower performance the model achieves (i.e., higher perplexity values). For instance, in the SEC dataset, the number of sensitive sentences in each category is as follows: 60 in GPE, 273 in location, 357 in PII, 1, 955 in organization, and 2, 166 in all entities. After 500 steps, the values of ϵ are 0.19 in GPE, 0.24 in location, 0.26 in PII, 0.73 in organization, 0.81 in all entities, and 4.08 in User-level DP (Fig. 4). At ϵ = 0.18 (Fig. 2c), we obtain perplexity values of 42.63 in GPE, 43.21 in location, 43.30 in PII, 43.70 in organization, 43.77 in all entities, and a 583.06 in User-level DP.
• Text classification. Fig. 3a shows that our UeDP-Alg achieves lower test error rates in terms of text classification on the AG dataset than baseline approaches in most cases across different types of sensitive entities under a very tight UeDP protection (ϵ ∈ [0.18, 0.19]). This is a promising result. When ϵ is higher, the test error rates of both UeDP-Alg and User-level DP drop, approaching the noiseless BERT model's upper-bound result.
• Extended Sensitive Entities. To shed light into the impact the extended sensitive entity sampling rate q s on model utility under UeDP protection, we varied the value of q s from 0 to 1 in all datasets and tasks. Figs. 3b, 5, and 7 show that considering extended sensitive entities (i.e., q s > 0) significantly improves model utility (i.e., perplexity or test error rate) compared with only considering sensitive entities e ∈ E (i.e., q s = 0). However, different tasks on different datasets may have different optimal values of q s . This opens a new research question on how to theoretically approximate the optimal value of q s .
Results on the AWD-LSTM model (Figs. 6 and7) further strengthen our observations. In our experiments, the AWD-LSTM model generally obtains comparable results with the GPT-2 model for next word prediction at a higher privacy budget range (i.e., ϵ ∈ [0.5, 3.0] in the AWD-LSTM model compared with ϵ ∈ [0.18, 0.2] in the GPT-2 model). This is because the GPT-2 model is pretrained on large-scale datasets, so that it is easily adapted to the idiosyncrasies of a target task (i.e., next word prediction) compared with the AWD-LSTM model trained from scratch.
In this paper, we developed a novel notion of user-entity DP (UeDP), protecting users' participation information and sensitive entities in NLMs. By incorporating user and sensitive entity sampling in the training process, we addressed the tradeoff between model utility and privacy loss with a tight bound of model sensitivity. Theoretical analysis and rigorous experiments show that UeDP-Alg outperforms baselines in next word prediction and text classification under UeDP protection.
In practice, the list of sensitive entities and users can grow over time. Periodically updating the list of users and sensitive entities may incur extra privacy and computational cost. Therefore, we will focus on preserving UeDP given a growing list of users and sensitive entities in our future work.
Authorized licensed use limited to: New Jersey Institute of Technology. Downloaded on June 20,2023 at 19:30:30 UTC from IEEE Xplore. Restrictions apply.
http://groups.di.unipi.it/ ∼ gulli/AG corpus of news articles.html
https://huggingface.co/transformers/pretrained models.html