Machine learning (ML) models (particularly deep neural networks) are vulnerable to inference attacks, which aim to infer sensitive information about the training data/dataset that are used to train the models. There are three well-known types of inference attacks on training data/dataset: membership inference attacks (MIAs) [12,58,73], property inference attacks (PIAs) (also called distribution inference attacks) [6,20,63], and data reconstruction attacks (DRAs) (also called model inversion attacks) [7,28]. Given an ML model, in MIAs, an adversary aims to infer whether a particular data sample was 1 Source code and the full version at: https://github.com/ leilynourbakhsh/Inf2Guard. in the training set, while in PIAs, an adversary aims to infer statistical properties of the training dataset used to train the targeted ML model. Furthermore, an adversary aims to directly reconstruct the training data in DRAs. Leaking the data sample or information about the dataset raises serious privacy issues. For instance, by performing MIAs, an adversary is able to identify users included in sensitive medical datasets, which itself is a privacy violation [30]. By performing PIAs, an adversary can determine whether or not machines that generated the bitcoin logs were patched for Meltdown and Spectre attacks [20]. More seriously, DRAs performed by an adversary leak all the information about the training data.
To mitigate the privacy risks, various defenses have been proposed against MIAs [35,45,54,56,58,59,61,71] and DRAs [21,25,38,48,55,62,69,81] 2 . However, there are two fundamental limitations in existing defenses: 1) They are designed against only one specific type of attack; 2) Provable defenses (based on differential privacy [3,18]) incur significant utility losses to achieve reasonable defense performance against inference attacks [33,56] since the design of such randomization-based defenses did not consider specific inference attacks (also see Section 5); and empirical defenses are soon broken by stronger/adaptive attacks [9,16,59].
We aim to address these limitations and consider the question: 1) Can we design a unified privacy protection framework against these inference attacks, that also maintain utility? 2) Under the framework, can we further theoretically understand the utility-privacy tradeoff and the privacy leakage against the inference attacks? To this end, we propose an information-theoretic defense framework, termed Inf 2 Guard, against inference attacks through the lens of representation learning [11]. Representation learning has been one of the biggest successes in modern ML/AI so far (e.g., it plays an important role in today's large language models such as Chat-GPT [1] and PaLM2 [2]). Particularly, rather than training large models from scratch, which requires huge computational costs and time (e.g., GPT-3 has 175 billion parameters), learning shared representations (or pretrained encoder) 3 presents an economical alternative. For instance, the shared representations can be directly used or further fine-tuned with different purposes, achieving considerable savings in time and cost.
More specifically, we formulate Inf 2 Guard via two mutual information (MI) 4 objectives in general, for privacy protection and utility preservation, respectively. Under this framework, we can design customized MI objectives to defend against each inference attack. For instance, to defend against MIAs, we design one MI objective to learn representations that contain as less information as possible about the membership of the training data-thus protecting membership privacy, while the other one to ensure the learnt representations include as much information as possible about the training data labels-thus maintaining utility. However, directly solving the MI objectives for each inference attack is challenging, since calculating an MI between arbitrary variables is often infeasible [49]. To address it, we are inspired by the MI neural estimation [4,10,15,29,47,50], which transfers the intractable MI calculations to the tractable variational MI bounds. Then, we are capable of parameterizing each bound with a (deep) neural network, and train neural networks to approximate the true MI and learn representations against the inference attacks. Finally, we can derive theoretical results based on our MI objectives: we obtain an inherent utility-privacy tradeoff, and guaranteed privacy leakage against each inference attack.
We extensively evaluate Inf 2 Guard and compare it with the existing defenses against the inference attacks on multiple benchmark datasets. Our experimental results validate that Inf 2 Guard obtains a promising utility-privacy tradeoff and significantly outperforms the existing defenses. For instance, under the same defense performance against MIAs, Inf 2 Guard has a 30% higher testing accuracy than the DP-SGD [3]. Our results also validate the privacy-utility tradeoffs obtained by Inf 2 Guardfoot_3 .
Our main contributions are summarized as below:
• Algorithm: We design the first unified framework Inf 2 Guard to defend against the three well-known types of inference attacks via information theory. Our framework can instantiate many existing defenses as special cases, e.g., AdvReg [45] against MIAs (See Section 3.1) and Soteria [62] against DRAs (See Section 3.3).
• Theory: Based on our formulation, we can derive novel theoretical results, e.g., the inherent tradeoff between utility and privacy, and guaranteed privacy leakage against all the considered inference attacks.
• Evaluation: Extensive evaluations verify the effectiveness of Inf 2 Guard for learning privacy-preserving representations against inference attacks.
Notations: We use s, s, S, and S to denote (random) scalar, vector, matrix, and space, respectively. Accordingly, Pr(s), Pr(s), and Pr(S) are the probability distribution over s, s, and S. I(x; r) and H(x, r) are the mutual information and cross entropy between a pair of random variables (x, r), respectively, and H(x) = I(x; x) as the entropy of x. KL(p||q) is the KLdivergence between two distributions p and q. We denote D
as the underlying distribution that data are sampled from. A data sample is denoted as (x, y) ⇠ D, where x 2 X is data features, y 2 Y is the label, and X and Y are the data space and label space, respectively. We further denote a dataset as D = {X, y} = {(x i , y i )}, that consists of a set of data samples
and will interchangeably use D and {X, y}. We let u 2 U be the private attribute within the attribute space U. For instance, in MIAs, u 2 U = {0, 1} means a binary-valued private membership; in PIAs, u 2 U = {1, 2, •• • , K} indicates a K-valued private dataset property; and u 2 U = X indicates the private data itself in DRAs. The composition function of two functions f and g is denoted as
We denote a classification modelfoot_4 F q : X ! Y as a function, parameterized by q, that maps a data sample x 2 X to a label y 2 Y . Given a training set D ⇠ D, we denote F T (D) as learned by running a training algorithm T on the dataset D.
Formalizing MIAs: Assume a data sample (x, y) ⇠ D with a private membership u that is chosen uniformly at random from {0, 1}, where u = 1 means (x, y) is a member of D, and 0 otherwise. An MIA A MIA has access to D and F, takes (x, y)
as input, and outputs a binary A D,F MIA (x, y). We omit D,F for notation simplicity. Then, the attack performance of an MIA A MIA is defined as Pr (x,y,u) (A MIA (x, y) = u). Then, the attack performance of a PIA A PIA is defined as
Formalizing DRAs: Given a random data (x, y) 2 D, DRAs aim to reconstruct the private x. A DRA A DRA has access to D and F, and outputs a reconstructed x = A DRA (x, y). The DRA performance is measured by the similarity/difference between x and x. For instance, [7] introduces the (h, g)-reconstruction metric defined as Pr (x,y) (kx xk 2 h) g, where a smaller h and a larger g imply a more severe DRA.
We have three roles: task learner, defender, and attacker. The task learner (i.e., data owner) aims to learn an accurate classification model on its training data. The defender (e.g., data owner or a trusted service provider) aims to protect the training data privacy-it designs a defense framework by learning shared data representations that are robust against inference attacks. The attacker can arbitrarily use data representations to perform the inference attack. The attacker is also assumed to know the underlying data distribution, but cannot access the internal encoder (e.g., deployed as an API [1,17]).
Formally, we denote f Q : X ! Z as the encoder, parameterized by Q, that maps a data sample x 2 X (or a dataset X 2 X ) to its representation vector r = f (x) 2 Z (or representation matrix R = f (X) 2 Z), where Z is the representation space. Moreover, we let C : Z ! Y be the classification model on top of the representation r or encoder f , which predicts the data label y (or dataset labels y). We further let A : Z ! U be the inference model, which infers the private attribute u using the learnt representations r or R. Then, our defense goals are:
• Defend against MIAs: Given a random sample (x, y, u) 2 D, we expect to learn f such that the MIA performance Pr(A MIA ( f (x), y) = u) is low, and the utility loss/risk, i.e., Risk MIA (C f ) = Pr(C f (x) 6 = y), is also small.
• Defend against PIAs: Given a random dataset (X, y, u) 2 D, we expect to learn f with low PIA performance Pr(A PIA ( f (X), y) = u), and also a small utility loss/risk, i.e., Risk PIA (C f ) = 1 |y| Â (x,y)2{X,y} Pr(C f (x) 6 = y). • Defend against DRAs: Given a random sample (x, y) 2 D, we expect to learn f with low DRA performance, i.e., Pr (x,y) (kx xk 2 h) g with a large h and g (flipping the inequality direction on h for DRAs), and also a small utility risk Risk DRA (C f ) = Pr(C f (x) 6 = y). or not (i.e., u = 0), the defender learns the representation r = f (x) that satisfies the following two goals:
• Goal 1: Membership protection. r contains as less information as possible about the private membership u. Ideally, when r does not include information about u (i.e., r ? u), it is impossible to infer u from r. Formally, we quantify the membership protection using the MI objective as follows:
where we minimize such MI to maximally reduce the correlation between r and u.
• Goal 2: Utility preservation. r should be effective for predicting the label y of the training data (i.e., u = 1), thus preserving utility. Formally, we quantify the utility preservation using the below MI objective:
where we maximize such MI to make r accurately predict the training data label y during training.
The key challenge of solving the above two MI objectives is that calculating an MI between two arbitrary random variables is likely to be infeasible [49].
Inspired by the existing MI neural estimation methods [4,10,15,29,47,50], we convert the intractable exact MI calculations to the tractable variational MI bounds. Specifically, we first obtain an MI upper bound for membership protection and an MI lower bound for utility preserving via introducing two auxiliary posterior distributions, respectively. Then, we parameterize each auxiliary distribution with a neural network, and approximate the true MI by minimizing the upper bound and maximizing the lower bound through training the involved neural networks. We emphasize we do not design novel MI neural estimators, but adopt existing ones to assist our MI objectives for learning privacy-preserving representations. Note that, though the estimated MI bounds may not be tight (due to the MI estimators or auxiliary distributions learnt by neural networks) [15, 29], they have shown promising performance in practice. It is still an active research topic to design better MI estimators that lead to tighter MI bounds (which is orthogonal to this work).
Minimizing the upper bound MI in Equation (1). We adapt the variational upper bound proposed in [15]. Specifically,
where q Y (u|r) is an auxiliary posterior distribution of p(u|r) needing to satisfy the below condition on KL divergence: KL(p(r, u)||q Y (r, u)) KL(p(r)p(u)||q Y (r, u)). To achieve this, we thus minimize:
where we note that E p(r,u) [log p(u|r)] is irrelevant to Y. [15] proved when q Y (u|r) is parameterized by a neural network with high expressiveness (e.g., deep neural network), the condition is satisfied almost surely by maximizing Equation (3). Finally, our Goal 1 for privacy protection is reformulated as solving the below min-max objective function:
[log q Y (u|r)] (4)
Remark. Equation ( 4) can be interpreted as an adversarial game between an adversary q Y (i.e., a membership inference classifier) who aims to infer the membership u from r; and the encoder f who aims to protect u from being inferred.
Maximizing the lower bound MI in Equation (2). We adopt the MI estimator proposed in [46] to estimate the lower bound of Equation (2). Specifically, we have [log q W (y|r, u = 1))]
where q W is an arbitrary auxiliary posterior distribution that aims to accurately predict the training data label y from the representation r. Hence, our Goal 2 for utility preservation can be rewritten as the following max-max objective function:
Remark. Equation ( 5) can be interpreted as a cooperative game between the encoder f and q W (e.g., a label predictor) that aims to preserve the utility collaboratively.
Objective function of Inf 2 Guard against MIAs. By combining Equations ( 4) and (5), our objective function of learning privacy-preserving representations against MIAs is:
where l 2 [0, 1] tradeoffs privacy and utility. That is, a larger l indicates a stronger membership privacy protection, while a smaller l indicates a better utility preservation.
In practice, we solve Equation (6) via training three parameterized neural networks (i.e., encoder f , membership protection network g Y associated with the posterior distribution q Y , and utility preservation network h W associated with the posterior distribution q W ) using data samples from the underlying data distribution. Specifically, we first collect two datasets D 1 and D 0 from a (larger) dataset, and they include the members and non-members, respectively. Then, D 1 is used for training the utility network h W (i.e., predicting labels for training data D 1 ) and the encoder f ; and both D 1 and D 0 are used for training the membership protection network g Y (i.e., inferring whether a data sample from D 1 /D 0 is a member or not) and the encoder f . With it, we can approximate the expectation terms in Equation ( 6) and use them to train the neural networks. Training the membership protection network g Y : We approximate the first expectation w.r.t. q Y as 7
where H(a, b) is the cross-entropy loss between a and b. Take a single data x with private u for example. The above equation is obtained by:
, where g Y ( f (x)) i indicates i-th entry probability, and q Y (u| f (x)) means the probability of inferring x's member u. The adversary maximizes this expectation aiming to enhance the membership inference performance.
Training the utility preservation network h W : We approximate the second expectation w.r.t. q W as:
We maximize this expectation to enhance the utility.
Training the encoder f : With the updated g Y and h W , the defender performs gradient ascent on Equation (6) to update f , which can learn representations that protect membership privacy and further enhance the utility. We iteratively train the three networks until reaching predefined maximum rounds. Figure 1 illustrates our Inf 2 Guard against MIAs. Algorithm 1 in Appendix details the training. [45]. We observe that AdvReg is a special case of Inf 2 Guard. Specifically, the objective function of AdvReg can be rewritten as:
where f : X ! [0, 1] |Y | now outputs a sample's probabilistic confidence score and g Y is a membership inference model aiming to distinguish between members and non-members.
Different from MIAs, PIAs leak the training data properties at the dataset-level. To align this, instead of using a random sample (x, y), we consider a random dataset (X, y) in PIAs. Specifically, let X = {x i } consist of a set of independent data samples and y = {y i } the corresponding data labels that are sampled from the underlying data distribution D; and X is associated with a private (dataset) property u.
Given a dataset X ⇠ D with a property u, the defender learns a dataset representation R = f (X) that satisfies two goalsfoot_5 :
• Goal 1: Property protection. R contains as less information as possible about the private dataset property u. Ideally, when R does not include information about u (i.e., R ? u), it is impossible to infer u from R. Formally, we quantify the property protection using the below MI objective:
• Goal 2: Utility preservation. R includes as much information as possible about predicting y. Formally, we quantify the utility preservation using the MI objective as below:
We estimate the bounds of Equations 7 and 8 as below.
Minimizing the upper bound MI in Equation ( 7). Following membership protection, Goal 1 is reformulated as solving the below min-max objective function:
where q Y (u|R) is an arbitrary posterior distribution.
Remark. Similarly, Equation ( 9) can be interpreted as an adversarial game between a property inference adversary q Y who aims to infer u from the dataset representations R and the encoder f who aims to protect u from being inferred.
Maximizing the lower bound MI in Equations (8). Similarly, we adopt the MI estimator [46] to estimate the lower bound MI in our Goal 2, which can be rewritten as the following max-max objective function:
where q W is an arbitrary posterior distribution that aims to predict each label y 2 y from the data representation r 2 R.
Remark. Equation ( 10) can be interpreted as a cooperative game between f and q W to preserve the utility collaboratively.
Objective function of Inf 2 Guard against PIAs. By combining Equations ( 9) and ( 10), our objective function of learning privacy-preserving representations against PIAs is:
where l 2 [0, 1] tradeoffs between privacy and utility. That is, a larger/smaller l indicates less/more dataset property can be inferred through the learnt dataset representation.
Equation ( 11) is solved via three parameterized neural networks (i.e., the encoder f Q , the property protection network g Y associated with q Y , and the utility preservation network h W associated with q W ) using a set of datasets sampled from a data distribution. Specifically, we first collect a large reference dataset D r . Then, we randomly generate a set of small datasets {D j = (X j , y j )} j from D r . We denote the dataset property value for each D j as u j . With it, we can approximate the expectation terms in Equation (11).
Training the property inference network g Y : We approximate the first expectation w.r.t.
where f (X j ) is the aggregated representation of a dataset X j , i.e., f (X j ) = Agg({ f (x)} x2X j ). We will discuss the aggregator Agg(•) in Section 5.2.2. The adversary maximizes this expectation to enhance the property inference performance.
Training the utility preservation network h W : Similarly, we approximate the second expectation w.r.t. q W as:
where we maximize this expectation to enhance the utility.
Training the encoder f : The defender then performs gradient ascent on Equation (11) to update f , which mitigates the PIA and further enhances the utility. We iteratively train the three networks until reaching maximum rounds. Figure 2 illustrates our Inf 2 Guard against PIAs. Algorithm 2 in Appendix details the training process.
Different from MIAs and PIAs, DRAs aim to directly recover the training data from the learnt representations. A recent defense [62] shows perturbing the latent representations can somewhat protect the data from being reconstructed. However, this defense is broken by an advanced attack [9]. One key reason is the defense perturbs representations in a deterministic fashion for already trained models. We address the issues and propose an information-theoretic defense to learn randomized representations against the DRAs in an end-toend learning fashion. Our core idea is to learn a deterministic encoder and a randomized perturbator that ensures learning the perturbed representation in a controllable manner.
Given a data sample x ⇠ D with a label y, the defender learns a representation r = f (x) such that when r is perturbed by certain perturbation (denoted as d d d), the shared perturbed representation r + d d d cannot be used to well recover x, but is effective for predicting y, from the information-theoretic perspective. Then we aim to achieve the following two goals:
•
• Goal 2: Utility preservation. To ensure r be useful, it should be effective for predicting the label y. Further, as we will share the perturbed representation r + d d d, it should be also effective for predicting y. Formally, we quantify the utility preservation using the MI objective as follows:
Minimizing the upper bound MI in Equation (14). Similarly, we adapt the variational upper bound in [15]. Our Goal 1 for data reconstruction protection can be reformulated as the below min-max objective function:
Remark. Equation ( 16) can be interpreted as an adversarial game between an adversary q Y (i.e., data reconstructor) who aims to infer x from r + d d d; and the encoder f who aims to protect x from being inferred via carefully perturbing r.
Maximizing the lower bound MI in Equation (15). Based on [50], we can produce a lower bound on the MI I(r; y) due to the non-negativity of the KL-divergence:
where q W is an arbitrary posterior distribution that predicts the label y from r and the entropy H(y) is a constant. We have a similar form for the MI I(r + d d d; y) as below
where we use the same q W to predict the label y from the perturbed representation r + d d d.
Then, our Goal 2 for utility preservation can be rewritten as the following max-max objective function:
Remark. Equation ( 19) can be interpreted as a cooperative game between the encoder f and the label prediction network q W , who aim to preserve the utility collaboratively.
Objective function of Inf 2 Guard against DRAs. By combining Equations ( 16)-( 19), our objective function of learning privacy-preserving representations against DRAs is:
where l 2 [0, 1] tradeoffs privacy and utility. A larger l implies less data features can be inferred through the perturbed representation, while a smaller l implies the shared perturbed representation is easier for predicting the label.
The key of our defense lies in defining the perturbation distribution p(d d d) in Equation (20). Directly specifying the optimal perturbation distribution is challenging. Motivated by variational inference [36], we propose to parameterize p(d d d) with trainable parameters, e.g., F. Then the optimization problem w.r.t. the perturbation d d d can be converted to be w.r.t. the parameters F, which can be solved via back-propagation.
A natural way to model the perturbation around a representation is using a distribution with an explicit density function.
Here we adopt the method in [36]
That is, it first samples u from a diagonal Gaussian with a mean vector µ µ µ and standard deviation vector s s s, and d d d is obtained by compressing u to be [ 1, 1] via the tanh(•) function and multiplying e. F = (µ µ µ, s s s) are the parameters to be learnt.
We train three neural networks (i.e., the encoder f , reconstruction protection network g Y , and utility preservation network h W ) using data samples from certain data distribution. Suppose we are given a set of data samples D = {x j , y j }. Learning the data reconstruction network g Y : As x and its representation r are often high-dimensional, the previous MI estimators are inappropriate in this setting. To address it, we use the Jensen-Shannon divergence (JSD) [29] specially for high-dimensional MI estimation. Assume we have updated F. We can approximate the expectation w.r.t. g Y as
where x 0 j is an independent and random sample from the same distribution as x j , and sp(z) = log(1 + exp(z)) is the softplus function. We maximize
Learning the utility preservation network h W : We first estimate the below expectation:
Similarly, we can approximate the third expectation as:
We minimize the two cross entropy losses to update h W .
Updating the distribution parameter F: Due to the reparameterization trick, the gradient can be back-propagated from each d d d j to the parameters F. For simplicity, we do not consider the JSD term in Equation ( 22) due to its complexity. Then we have the terms relevant to F as below:
where b = la/(1 l). The first term is the cross entropy loss, while the second term is the entropy. The gradient w.r.t. F in each term can be calculated. In practice, we approximate the expectation on z z z with (e.g., 5) Monte Carlo samples, and perform the stochastic gradient descent to update F. Details on updating F are in Algorithm 3. With F, we use it to generate d d d and add it to r to produce the perturbed representation.
Learning the encoder f . Finally, after updating g Y , h W , and F, we can perform gradient ascent to update f . We iteratively train the networks until reaching a predefined maximum round. Figure 3 illustrates Inf 2 Guard against DRAs. Algorithm 4 in Appendix details the training process.
Due to limited space, we mainly show the guaranteed privacy leakage under Inf 2 Guard. We also derive an inherent utility-privacy tradeoff of Inf 2 Guard, which requires a binary classification task, and binary-valued dataset property in PIAs. In practice, once the encoder f is learnt on a dataset from D, I(u; r) can be estimated, then the bounded MIA accuracy can be calculated. A better encoder f or/and better MI estimator of I(u; r) can yield a smaller MIA performance. Remark. Theorem 2 shows when H(u|R) is larger, the PIA accuracy is smaller, i.e., less dataset property is leaked. Also, a large H(u|R) indicates a small I(u; R)-This is exactly our Goal 1 in Equation ( 7) aims to achieve.
In this section, we will evaluate Inf 2 Guard against the MIAs, PIAs, and DRAs on benchmark datasets. Inf 2 Guard involves training the encoder, the privacy protection network, and the utility preservation network. The detailed dataset description and architectures of the networks are given in the full version.
Datasets: Following existing works [35,45], we use the CI-FAR10 [37], Purchase100 [45], and Texas100 [58]
Utility-privacy results: According to Equation ( 6), l = 0 indicates no privacy protection. Increasing l's value enhances Inf 2 Guard's resilience against MIAs. l = 1 means the maximum privacy protection without preserving utility. Table 1 shows the utility-MIA Accuracy results of Inf 2 Guard. We have the following observations: 1) The MIA accuracy is the largest when l = 0, implying leaking the most membership privacy by MIAs. 2) When only protecting privacy (l = 1), the MIA accuracy reaches to the optimal random guessing, but the utility is the lowest. 3) When 0 < l < 1, Inf 2 Guard obtains reasonable utility and MIA accuracy. Especially, when l = 0.75, the utility loss is marginal (i.e., < 4%), while the MIA accuracy is (close to) random guessing. The results show the learnt privacy-preserving encoder/representations are effective against MIAs, and maintain utility as well. Further, Figure 4 shows the TPR vs FPR of Inf 2 Guard against LiRA. Similarly, we observe that the TPR at low FPRs is relatively large (strong membership inference) in case of no privacy protection, but it can be largely reduced by increasing l. This implies that Inf 2 Guard indeed learns the representations that can defend against LiRA to some extent. (a) Utility w/o. defense (49.8%) (b) MIA Acc w/o. defense (70.2%) (c) Utility w. defense (46%) (d) MIA Acc w. defense (50%) Figure 7: Inf 2 Guard against MIAs: 3D t-SNE embeddings results on the learnt representation on Texas100.
Visualizing the learnt representations: To better understand the learnt representations by Inf 2 Guard, we adopt the t-SNE algorithm [64] to visualize the low-dimensional embeddings of them. l is chosen in Table 1 that achieves the best utilityprivacy tradeoff. We also compare with the case without privacy protection. Figures 567show the 3D t-SNE embeddings, where each color corresponds to a label in the learning task or (non)member in the privacy task, and each point is a data sample. We can observe the t-SNE embeddings of the learnt representations without privacy protection for members and non-members are separated to some extent, meaning the membership can be inferred via the learnt MIA classifier. On the contrary, the t-SNE embeddings of the learnt representations by our Inf 2 Guard for members and non-members are mixedhence making it difficult for the (best) MIA classifier to infer the membership from these learnt representations.
Comparing with the existing defenses against MIAs: All empirical defenses are broken by stronger attacks [16,59], except adversarial training-based AdvReg [45] (a special case
Table 2: Comparing Inf 2 Guard with existing defenses against MIAs on the three datasets. DP methods are under the same/close defense performance as Inf 2 Guard. Defense CIFAR10 Purchase100 Texas100 Utility MIA Acc Utility MIA Acc Utility MIA Acc DP-SGD 48% 51% 40% 52% 11% 51% DP-enc 45% 51% 32% 51% 10% 50% AdvReg 75% 53% 75% 51% 44% 52% NeuGuard 74% 56% 77% 53% 43% 52% Inf 2 Guard 77% 51% 80% 51% 46% 50%
of Inf 2 Guard). NeuGuard [71] is a recent empirical defense and shows better performance than, e.g., [35,57]. Differential privacy is the only defense with privacy guarantees. We propose to use two DP variants, i.e., DP-SGD [3] and DPencoder (details in Appendix A). The comparison results of these defenses are shown in
Table 2 (more DP results in Table 12 in Appendix B) and Figure 8. From Table 2, we observe DP methods have bad utility when ensuring the same level defense performance (w.r.t. MIA accuracy) as Inf 2 Guard. AdvReg and NeuGuard also perform worse than Inf 2 Guard. Figure 8 shows the TPR vs. FPR of these defenses against LiRA under the results in Table 2. For DP methods, we also plot the TPR vs FPR when their utility is close to Inf 2 Guard. With an MIA accuracy close to random guessing (but low utility), we see DP methods have the smallest TPR at a given low FPR. This means DP methods can most reduce the attack effectiveness of LiRA, which is also verified in [12]. However, if DP methods have a close utility as Inf 2 Guard, their TPRs are much higher than Inf 2 Guard's at a low FPR. Besides, Inf 2 Guard has smaller TPRs than AdvReg and NeuGuard.
Overhead comparison: All MIA defenses train a task classifier. AdvReg trains a task classifier and membership inference network. NeuGuard trains a task classifier with two regularizations. DP-SGD trains the task classifier on noisy models, while DP-encoder normally trains the encoder first and then trains the utility network on (Gaussian) noisy representations.
In the experiments, we define the task classifier of the compared defenses as the concatenation of our encoder and utility network. In our platform (NVIDIA GeForce RTX 3070 Ti), it took Inf 2 Guard (72,7,6), AdvReg (66,6,6), NeuGuard (62,5,5), DP-SGD (60,4,3) and DP-encoder (59,4,3) seconds to run each iteration on the three datasets, respectively 9 .
Datasets: Following recent works [13,63], we use three datasets (Census [63], RSNA [63], and CelebA [40]) and treat the female ratio as the private dataset property.
Utility-privacy results: Table 3 shows the utility-privacy results of Inf 2 Guard, where the encoder uses a meanaggregator (i.e., average the representations of a subset of data. Note different subsets have different sizes). We have similar observations as in defending against MIAs: 1) The PIA accuracy can be as large as 68% without privacy protection (l = 0 in Equation ( 11)), implying the PIA is effective; 2) When focusing on protecting privacy (l = 1), the PIA performance can be largely reduced. However, the utility is also significantly decreased, e.g., from 85% to 45%. 3) Utility and privacy show a tradeoff w.r.t. 0 < l < 1. In most of the cases, the best tradeoff is obtained when l = 0.75. Again, the results show the learnt privacy-preserving encoder/representations are effective against PIAs, and also maintain utility. Impact of the aggregator used by the encoder: In this experiment, we test the impact of the aggregator and choose a max-aggregator for evaluation, where we select the elementwise maximum value of the representations of each subset of data. Table 4 shows the results. We have similar conclusions as those with the mean-aggregator. In addition, Inf 2 Guard with the max-aggregator has slightly worse utility-privacy tradeoff, compared with the mean-aggregator. A possible reason could be the mean-aggregator uses more information of the subset representations than the max-aggregator.
Comparing with the DP-based defense: There exists no effective defense against PIAs, and [63] shows DP-SGD [3] does not work well. Here, we propose to use a DP variant called DP-encoder, similar to that against MIAs. More details about DP-encoder are in Appendix A. The compared results are shown in Table 5. We can see that, with the same level privacy protection as Inf 2 Guard, DP has much worse utility.
Datasets: We select two image datasets: CIFAR10 [37] and CIFAR100 [37], and one human activity recognition dataset Activity [51] to evaluate Inf 2 Guard against DRAs.
Defense/attack training and testing: Table 11 in Appendix shows the statistics of the utility/attack training and test sets.
• Defense training: We use the training set to train the encoder, utility preservation network, reconstruction protection network, and update the perturbation distribution parameters, simultaneously. Then, the learnt encoder and perturbation distribution are published. • Attack training: We mimic the strongest DRA, where the attacker knows the reconstruction protection network, training set, and perturbation distribution. S/he feeds each training data to the learnt encoder + perturbation distribution to get the perturbed representation. Then the attacker trains the reconstruction network (using the pair of input data and its perturbed representation) to infer the training data.
• Defense/attack testing: We use the utility test set to obtain the utility via querying the encoder and utility network; and use the attack test set to obtain the DRA performance by querying the trained encoder and reconstruction network.
Privacy metric: For image datasets, we use the common Structural Similarity Index Measure (SSIM) and PSNR metrics [27]. A larger SSIM (or PSNR) between two images indicate they look more similar. An effective attack aims to achieve a large SSIM (or PSNR), while the defender does the opposite. For human activity dataset, we use the mean-square error (MSE) between two samples to measure similarity. A smaller/larger MSE indicates a more effective attack/defense.
Utility-privacy results: Table 6 shows the defense results of Inf 2 Guard with the Gaussian perturbation distribution, where l = 0.4 in Equation (20). We can observe e acts a utilityprivacy tradeoff. A larger e implies adding more perturbation to the representation during defense training. This makes the DRA more challenging, but also sacrifice the utility more. We also test the impact of l and the results are shown in Table 7. We can see l also acts as a tradeoff-a larger l can protect data privacy more, while having larger utility loss.
Comparing with the DP-based defense: All empirical defenses against DRAs are broken are by an advanced attack [9]. A few papers [7,53] show if a randomized algorithm satisfies DP, it can defend against DRAs with provable guarantees. We compare Inf 2 Guard with DP and Table 8 shows the DP results. Viewing with results in Table 6, we see Inf 2 Guard obtains better utility-privacy tradeoffs than DP-SGD. Visualizing data reconstruction results: Figure 9 and Figure 10 show the reconstruction results on some CIFAR10 and CIFAR100 images, respectively. We see that, without defense, the attacker can accurately reconstruct the raw images. With a similar utility, visually, Inf 2 Guard can better defend against image reconstruction than DP. Figure 11 summarizes the reconstruction results on 50 samples in Activity, where we report the difference between each reconstructed feature by Inf 2 Guard and that by DP to the true feature. A (larger) positive value implies Inf 2 Guard is (more) dissimilar than DP to the true feature. We can see Inf 2 Guard has better defense results than DP in most (413 out of 516) of the features.
6 Discussion and Future Work
Inf 2 Guard and DP: Essentially, Inf 2 Guard and DP are two different provable privacy mechanisms, and they complement each other. First, DP mainly measures the user or sample-level privacy risks in the worst case while Inf 2 Guard can accurately measure the average privacy risks at the dataset level with the derived bounds. Second, DP has been shown to provide some resilience transferability across some inference attacks [53] (but not all of them). It is also interesting to study the resilience transferability for the proposed Inf 2 Guard, which we will explore in the future. More importantly, our Inf 2 Guard can complement DP. For instance, we can use the learnt (deterministic) data representations by Inf 2 Guard as input to DP-SGD or add (Gaussian) noise to the representations to ensure DP guarantees against MIAs.
Task-agnostic representation learning: Our current MI formulation for utility preservation knows the labels of the learn- ing task (e.g., see Equation ( 2)). A more promising solution would be task-agnostic, i.e., learning task-agnostic representations that can benefit many (unknown) downstream tasks.
We note that our framework can be easily extended to this scenario. For instance, in MIAs, we now require the learnt representation r includes as much information about the training sample x as possible (i.e., u = 1). Intuitively, when r retains all information about x, the model trained on r will have the same performance as trained on the raw x, despite the learning task. Formally, the MI objective becomes max f I(x; r|u = 1).
Defending against multiple inference attacks simultaneously: We design the customized MI objectives to defend against each inference attack in the paper. A natural solution to defend against multiple inference attacks is unifying their training objectives (by summarizing them with tradeoff hyperparameters). While this is possible, we emphasize that the learnt encoder is weak against all attacks. This is because the encoder should balance the defense effectiveness among these attacks, and cannot be optimal against all of them.
Generalizing our theoretical results: Our theoretical results assume the learning task is binary classification and dataset property is binary-valued. We will generalize our theoretical results to multiclass classification and other types of learning such as regression and multi-valued dataset property.
Generalizing our framework against security attacks: In our current framework, each privacy protection task is formalized via an MI objective. An important future work would be generalizing our framework to design customized MI objectives to learn robust representations against security attacks such as evasion, poisoning, and backdoor attacks.
7 Related Work 7.1 MIAs and Defenses MIAs [12, 14, 16, 31, 39, 54, 58-60, 70, 73, 76]. Existing MIAs can be classified as training based [12,14,16,39,52,54,58,60, 72, 73] and non-training based [16, 59]. Given a (non)training sample and its output by a target ML model, training based MIAs use the (sample, output) pair to train a binary classifier, which is then used to determine whether a testing sample belongs to the training set or not. For instance, [58] introduces multiple shadow models to perform training. In contrast, non-training based MIAs directly use the samples' predicted score/label to make decisions. For instance, [59] designs a metric prediction correctness, which infers the membership based on whether a given sample is correctly classified by the target model or not. Overall, an MIA that has more information is often more effective than that has less information.
Defenses [35,45,54,56,58,59,61,71]. They can be categorized as training time based defense (e.g., dropout [54], L 2 norm regularization [58], model stacking [54], adversary regularization [45], loss variance deduction [71], DP [3,32,75], early stopping [59], knowledge distillation [56]) and inference time based defense (e.g., MemGuard [35]). Almost all of them are empirical and broken by stronger attacks [16,59]. DP is only defense offering privacy guarantees. Its main idea is to add noise to the gradient [3,75] or objective function [32] during training. The main drawback of current DP methods is that they have significant utility losses [33,56].
PIAs [5,6,13,20,24,41,42,63,66,77,79]. Ateniese et al. [6] are the first to describe the problem of the PIA (against support vector machines and hidden Markov models), where the attack is performed in the the white-box setting and consists of training a meta-classifier on top of many shadow models. Ganju et al. [20] extend PIAs to neural networks, particularly fully connected neural networks (FCNNs). Zhang et al. [77] propose PIAs in the black-box setting and train a meta-classifier based on shadow models. Mahloujifar et al. [41] observe that data poisoning attacks can be incorporated into training the shadow model and increase the effectiveness of PIAs. Suri and Evans [63] are the first to formally formalize PIAs as a cryptographic game, inspired by the way to formalize MIAs [73].
They also extend the white-box attack on FCNNs [20] to convolutional neural networks (CNNs). Zhou et al. [79] develop the first PIA against generative models, i.e., generative adversarial networks (GANs) [23], under the black-box setting. Chaudhari et al. [13] propose a data poisoning strategy to perform the efficient private property inference.
Defenses. To our best knowledge, there exist no known effective defenses against PIAs. DP cannot mitigate PIAs since it obfuscates individual samples, while PIAs care about the entire datasets [63]. [63] also shows that DP does not work as a potential defense (also verified in Section 5).
DRAs [7][8][9]19,22,27,28,34,67,68,74,78,81]. Existing DRAs mainly reconstruct the training data from the model parameters or representations. They are formulated as an optimization problem that minimizes the difference between gradient from the raw training data and that from the reconstructed data.
For instance, Zhu et al. [81] proposed a DLG attack method which relies entirely on minimization of the difference of gradients. Furthermore, several methods [22,28,34,67,74] propose to incorporate prior knowledge (e.g., total variation regularization [22,74], batch normalization statistics [74]) into the training data, or introduce an auxiliary dataset to simulate the training data distribution [28,34,67] (e.g., via GANs [23]). A few works [22,80] derive close-formed solutions to reconstruct the data, by constraining the neural networks to be fully connected [22] or convolutional [80].
Defenses [21,25,38,48,55,62,69,81]. Most of these defenses have none/little privacy guarantees. For instance, Zhu et al. [81] propose to prune model parameters with smaller magnitudes. Sun et al. [62] propose to obfuscate the gradient for a single layer (called defender layer) such that the reconstructed data and the original data are dissimilar. Gao et al. [21] propose to generate augmented images that, when they are used to train the network, produce non-invertible gradients. These defenses are broken by an advanced attack based on Bayesian learning [9]. Only defenses based on DP-SGD [3], a version of SGD with clipping and adding Gaussian noise, provide formal privacy guarantees.
We propose a unified information-theoretic framework, dubbed Inf 2 Guard, to learn privacy-preserving representations against the three major types of inferences attacks (i.e., membership inference, property inference, and data reconstruction attacks). The framework formalizes the utility preservation and privacy protection against each attack via customized mutual information objectives. The framework also enables deriving theoretical results, e.g., inherent utility-privacy tradeoff, and guaranteed privacy leakage against each attack. Extensive evaluations verify the effectiveness of Inf 2 Guard for learning privacy-preserving representations and show the superiority over the compared baselines.
Algorithm 1 Inf 2 Guard against MIAs Input: Dataset D 1 of members and dataset D 0 of non-members, tradeoff hyperparameter l 2 [0, 1], learning rates lr 1 , lr 2 , lr 3 ; #local gradients I, #global rounds T . Output: Network parameters: Q, Y, W. 1: Initialize Q, Y, W for the encoder f , membership protection network g Y , and utility preservation network h f ; 2: for t = 1 to T do 3:
Algorithm 2 Inf 2 Guard against PIAs
Input: N datasets {D j } N j=1 sampled from a reference dataset D r with each D j having a property value u j , tradeoff hyperparameter l 2 [0, 1], learning rates lr 1 , lr 2 , lr 3 ; #local gradients I, #global rounds T .
Output: Network parameters: Q, W, Y.
1: Initialize Q, Y, W for the encoder f , property protection network g Y , and utility preservation network h f ; 2: for round t = 1 to T do 3:
Algorithm 3 Update perturbation distribution parameter F Input: K Monte Carlo samples, the encoder f Q in the previous round, objective function Eqn (20). learning rate lr, #epochs I l
Output: Perturbation distribution parameters F 1: Initialize F = (µ µ µ, s s s).
2: for i = 1 to I l do 3: for j = 1 to K do 4: Sample z j from N (0, 1) and compute d d d j = µ µ µ + s s sz z z j ;
Calculate the gradient g F of Eqn (25) w.r.t. F; 6: Update F by: F F lr • g F . Algorithm 4 Inf 2 Guard against DRAs Input: A dataset D = {x n , y n }, hyperparameters l 2 [0, 1], learning rates lr 1 , lr 2 , lr 3 , #local gradients I, #global rounds T . Output: Network parameters: W, Y, Q. 1: Initialize Q, Y, W, F for the encoder f , data reconstruction network g Y , utility preservation network h W , and perturbation distribution parameter. 2: for round t = 1 to T do 3: for each batch bs ⇢ D do 4: Update F via Algorithm 3; 5: Update g Y (given Q and {d d d i }): Calculate I (JSD) Q,Y on bs with {d d d i } via Eqn (22); Y Y + lr 1 • ∂I (JSD) Q,Y /∂Y; 6: Update h W (given Q and {d d d i }): Calculate CE loss L 1 on bs with {d d d i } via Eqn (23); Calculate CE loss L 2 on bs with clean data via Eqn (24); W W lr 2 • ∂(L 1 + L 2 )/∂W; 7: Update f Q (given Y, W, and {d d d Table 9-Table 11 show the utility training/test and attack training/test sets on the three datasets. Differential Privacy (DP) against MIAs: DP provides an upper bound on the success of any MIA. We can add noise in several ways (e.g., to input data, model parameters, gradients, latent features, output scores) to ensure DP. Note that there exists an inherent trade-off between utility and privacy: a larger added noise often leads to a higher level of privacy protection, but incurs a larger utility loss. Here, we propose to use the below two ways. • DP-SGD [3]: 1) DP-SGD training: It clips gradients (with a gradient norm bound) and adds Gaussian noise to the gradient in each SGD round when training the ML model (i.e., encoder + utility network). More details can be seen in Algorithm 1 in [3]. After training, the model ensures DP guarantees and the encoder is published. 2) Attack training: The attacker obtains the representations of the attack training data via querying the trained encoder and uses these representations to train the MIA classifier. 3) Defense/attack testing: The utility test set is used to obtain the utility via querying the trained ML model; and the attack test set to obtain the MIA accuracy via querying the trained encoder and trained MIA classifier.
We used the Opacus library (https://opacus.ai/), a Py-Torch extension that enables training models with DP-SGD and dynamically tracks privacy budget and utility. In the experiments, we tried e in DP-SGD from 0.5 to 16.
• DP-encoder: 1) Normal training: It first trains the encoder + utility network using the (utility) training set. The encoder is then frozen and can be used to produce data representations when queried by data samples. 2) Defense via adding noise to the representations: We add Gaussian noise to the representations by querying the encoder with the attack training data to produce the noisy representations. Notice that, since the Gaussian noises are injected to the matrixoutputs (data representations), if needed, the actual DP guarantee (i.e., privacy bounds) can be derived via Rényi Differential Privacy [43], similar to the theoretical studies in [44,65]. We skip the details here since this work does not focus on the derivation for the privacy bounds of DP-encoder. DP-encoder against PIAs: We follow the strategy in DP against MIAs and choose the DP-encoder, as DP-SGD is ineffective in this setting [63]. The only difference is that we now add Gaussian noise to the mean-aggregated representation of a subset, instead of the individual representation.
Data reconstruction attack/defense on shallow encoder: As shown in [27], when the encoder is deep, it is difficult for the attacker to reconstruct the input data from the representation. To ensure DRAs be effective, we use a shallow 2-layer encoder. As a result, this makes the defense more challenging.
More results on defending against MIAs: Table 12 shows more DP results (vs varying e's) against MIAs. We can see Inf 2 Guard obtains higher utility than DP methods under the same privacy protection performance.
More results on defending against PIAs: Figure 12-Figure 14 shows the t-SNE embeddings of Inf 2 Guard against PIAs. Table 13 shows Inf 2 Guard results against PIAs, where the attacker does not the true (mean) aggregator and use a substitute one (i.e., max-aggregator ). We can see the attack performance is less effective and Inf 2 Guard can yield (close to) random guessing attack performance (when g = 0.75), with a slight utility loss. This implies the aggregator plays a critical role in designing effective PIAs against Inf 2 Guard. More results on defending against DRAs: Table 14 shows the Inf 2 Guard results against DRAs, where the perturbation distribution is uniform distribution. We observe similar utilityprivacy tradeoff in terms of the noise scale e.
To our best knowledge, there exist no effective defenses against PIAs.[26] analyzes sources of information leakage to cause PIAs, but their solutions are difficult to be tested on real-world datasets due to lack of generality.
Pretrained encoder as a service has been widely deployed by industry, e.g., OpenAI's GPT-4 API [1] and Clarifai's Embedding API[17]. We will interchangeably use the pretrained encoder and learnt representations.
In information theory, MI is a measure of shared information between random variables, and offers a metric to quantify the "amount of information" obtained about one random variable by observing the other random variable.
A recent work[53] formulates defenses against inference attacks under a privacy game framework, but it does not propose concrete defense solutions.
In this paper, we focus on classification models for simplicity.
For notation simplicity, we use the same f to indicate the encoder. Similar for subsequent notations such as g Y , h Y , q W , h W , etc.
We have similar conclusions on defending against the other two attacks.