Do Parameters Reveal More than Loss for Membership Inference?

Introduction

Models produced by using machine learning on private training data can leak sensitive information about data used to train or tune the model (Salem et al., 2023). Researchers study these privacy risks by either designing and evaluating attacks to simulate what motivated adversaries may be able to infer in particular settings or by developing privacy methods that provide strong guarantees, often based on some notion of differential privacy (Dwork et al., 2006), that bounds information disclosure from any attack. Although both developing attacks and formal privacy proofs are important, conducting meaningful privacy audits is different from both approaches. Empirical methods, usually in the form of attack simulations, are inherently limited by the attacks considered and the uncertainty about the possibility of better attacks, while theoretical proofs require many assumptions or result in loose bounds. Further, any claims based on theoretical results depend on careful analysis that the system as implemented is consistent with the theory. If there is a theoretical result that prescribes an optimal attack, then empirical results with that attack (or approximations of the attack) can offer a more meaningful estimate of privacy risk than is possible with theory or experiments alone. While the theory needs to cover all data distributions, experiments with an optimal attack focus on the actual distribution and given model, resulting in tighter and more relevant privacy evaluations.

Privacy audits can also be important in adversarial contexts, where a regulator or external advocate conducts them to test a released model. Auditors with elevated model access (such as associated training environments or data) may be able to take advantage of more information to produce better estimates of what an adversary could do without that information. Auditing is orthogonal to proofs that establish differential privacy bounds or other privacy notions. As outlined by Cummings et al. (2024), theoretical bounds may be "too conservative or inaccurate in some settings", and it may not always be possible to come up with proofs or theoretical bounds that ensure models do not "violate disclosure requirements in ways that are not captured by differential privacy". Empirical auditing can provide a more meaningful measure of privacy leakage for these situations.

The most common disclosure auditing approach today is to conduct membership inference attacks (Kumar & Shokri, 2020) and related attacks that attempt to extract specific data (Cummings et al., 2024). While membership inference assumes the adversary already knows the full candidate record, it may still constitute a direct privacy risk when revealing the inclusion of a known record in the training data itself, which leaks sensitive information. In most scenarios, however, membership disclosure by itself is not a serious privacy risk, but rather used as a proxy for understanding information leakage that may result in more serious privacy violations. Membership inference is simple to define, relatively easy to measure, and aligns well with differential privacy. This has resulted in it being widely used as a method for auditing disclosure risks for machine learning (Kumar & Shokri, 2020;Yeom et al., 2020;Kazmi et al., 2024;Azize & Basu, 2024).

Prior results on membership inference attacks have largely focused on the black-box setting, where the attacker only has input-output access to the target model. This focus has been reinforced by folklore and results demonstrating negligible gains from parameter access (known as white-box attacks) (Nasr et al., 2018;Carlini et al., 2022). A well-known theoretical result by Sablayrolles et al. (2019) proves that black-box access is sufficient for optimal membership inference under certain conditions. This result has been the basis of several subsequent works (Ye et al., 2022;Chaudhari et al., 2024). However, the assumptions made in its derivation do not hold for most models, including ones trained with stochastic gradient descent (SGD). This theoretical result has detered researchers from exploring more the potential for inference methods that utilize parameter access, even though the theoretical result does not apply to common machine learning settings.

Membership Inference

Following the framework established by Sablayrolles et al. (2019), let D be a data distribution from which n records z 1 , z 2 , . . . , z n are i.i.d. sampled with z i = (x i , y i ) being the i-th record. Let w ∈ R d be the model parameters produced by some machine learning algorithm on a training dataset D. Assume m 1 , m 2 , . . . , m n follow a Bernoulli distribution with γ = P(m i = 1), where m i is the membership indicator of z i (i.e., m i = 1 if z i ∈ D, and m i = 0 otherwise). Given w, a membership inference attack aims to predict the unknown membership m i for any given record z i . Definition 2.1 (Membership Inference). Let w be the parameters of the target model and z 1 be a record. Inferring the membership of z 1 to the training set of w is equivalent to computing:

Let P(w | z 1 , . . . , z n , m 1 , . . . , m n ) be the posterior distribution of model parameters produced by some randomized machine learning algorithm (i.e., stochastic gradient descent). Applying Bayes' theorem, Sablayrolles et al. (2019) derived the following explicit formula for M(w, z 1 ). Lemma 2.1 (Sablayrolles et al. (2019)). Let T = {z 2 , . . . , z n , m 2 , . . . , m n }. Given model parameters w and a record z 1 , the optimal membership inference is given by:

where σ(u) = (1 + exp(-u)) -1 is the Sigmoid function, and γ = P(m 1 = 1).

To use Lemma 2.1, one needs to characterize the posterior, P(w | z 1 , . . . , z n , m 1 , . . . , m n ), to make explicit the effect of the inferred record z 1 on the optimal membership inference M(w, z 1 ). Recent advances in discrete-time SGD dynamics (Liu et al., 2021;Ziyin et al., 2021) literature can help provide a connection between the posterior and model parameters.

Discrete-time SGD Dynamics

A line of theoretical work (Welling & Teh, 2011;Sato & Nakagawa, 2014;Stephan et al., 2017;Liu et al., 2021;Ziyin et al., 2021) has analyzed the continuous-and discrete-time dynamics of stochastic gradient methods and provided insights for understanding deep learning generalization. Let L tot (w) = L(w) + α 2 ∥w∥ 2 2 be the ℓ 2 -regularized total loss that we aim to optimize, where α ≥ 0 denotes the hyperparameter that controls the regularization strength. Consider an SGD algorithm with the following update rule (for t = 1, 2, 3, . . .):

(2)

Here, µ ∈ [0, 1) is the momentum, λ > 0 is the learning rate, and

represents the unbiased mini-batch noise, where B t is a randomly sampled batch of examples with size S from the training dataset D, and L(w) = 1 n z∈D ℓ(w, z). Assuming a model is trained using SGD according to the update rule defined by Equation 2 on a quadratic loss and arrives at a stationary state, Liu et al. (2021) established a theoretical connection between the Hessian matrix H, the asymptotic noise covariance C = lim t→∞ E wt [cov(η t , η t )], and the asymptotic model fluctuation Σ = lim t→∞ cov(w t , w t ). We next lay out the two imposed assumptions. Assumption 1 (Quadratic Loss). The total loss function L tot (w) is either globally quadratic or locally quadratic close to a local minimum w * . Specifically, the loss function can be approximated as:

where w * is a local minimum, H(w * ) denotes the Hessian matrix at w * with respect to the unregularized loss function L(w), and I d stands for the d × d identity matrix. Assumption 2 (Stationary-State). After a sufficient number of iterations, models trained with SGD defined by Equation 2 arrive at a stationary state, i.e., the asymptotic model fluctuation Σ exists and is finite.

Under the above assumptions, Liu et al. (2021) proved the following theorem that describes model fluctuations of discrete SGD in a quadratic potential with connections to the Hessian matrix and the noise covariance: Theorem 2.2 (SGD Stationary distribution with momentum). Let w be updated with SGD defined by the update rule in Equation 2 with momentum µ ∈ [0, 1). Under Assumptions 1 and 2, if we additionally suppose C commutes with H(w * ), then the asymptotic model fluctuation satisfies

Theorem 2.2 requires the existence of a finite stationary noise covariance and that the loss function is quadratic close to a local minimum, which are both mild assumptions (see Liu et al. (2021) for detailed discussions).

In a follow-up work, Ziyin et al. ( 2021) further derived the explicit dependence of the finite stationary noise covariance C on the loss and Hessian around a local minimum w * under certain assumptions:

Theorem 2.3 (SGD Noise Covariance). Let L tot (w) = L(w) + α 2 ∥w∥ 2 2 be the total loss with α ≥ 0. Assume the model w is optimized with SGD defined by Equation 2 around a local minimum w * . If L(w * ) ̸ = 0, then

provided that Σ is proportional to S -1 and |L(w) -ℓ(w, z i )| is small (i.e., of order o(L(w)).

The first imposed assumption of Σ = O(S -1 ) has been justified by prior works (Liu et al., 2021;Xie et al., 2021;Mori et al., 2022); the second assumption assumes that the current total training loss L(w) approximates well the individual loss for each record ℓ(w, z i ). Also, note that Theorem 2.3 directly implies that the SGD noise covariance C commutes with the Hessian matrix H(w * ).

Based on the above two theorems and only considering the leading term in the noise covariance, we can immediately derive the following formula for the stationary model fluctuation of SGD:

We remark that if L(w * ) = 0 (i.e., w * is a global minimum), then Σ = 0. In addition, if the Hessian matrix (H(w * ) + αI d ) has degenerate rank r < d, then (H(w * ) + αI d ) -1 can be replaced by the corresponding Moore-Penrose pseudo inverse. Accordingly, similar results to Equation 4 can be obtained by considering the projection space spanned by eigenvectors with non-zero eigenvalues. Section 5 of Ziyin et al. (2021) provides more detailed discussions of the imposed assumptions and the implications of the results.

Limitations of Claims of Black-Box Optimality

Sablayrolles et al. (2019) proved the optimality of black-box membership inference under a Bayesian framework. They assume (Equation 1 in Sablayrolles et al. (2019)) that the posterior distribution of model parameters w trained on z 1 , . . . , z n with membership m 1 , . . . , m n follows

where T is a temperature parameter that captures the stochasticity of the learning algorithm. This assumption makes subsequent derivations of optimal membership inference much easier, but oversimplifies the training dynamics of typical machine learning algorithms such as SGD. Equation 5 assumes that the posterior of w follows a Boltzmann distribution that only depends on the training loss. This is desirable for Bayesian posterior inference, where the goal is to provide a sampling strategy for an unknown data distribution given a set of observed data samples. This can be achieved using stochastic gradient Langevin dynamics (SGLD) (Welling & Teh, 2011) with shrinking step size λ t (i.e., lim t→∞ λ t = 0) and by injecting carefully-designed Gaussian noise N (0, λ t • I D ). However, this special SGLD design differs from the common practice of SGD algorithms used to train neural networks in two key ways:

1. SGLD performs all analyses under continuous-time dynamics whereas actual SGD is performed with discrete steps. While related work such as Stephan et al. (2017) cast the continuous-time dynamics of SGD as a multivariate Ornstein-Uhlenbeck process (similar to SGLD) whose stationary distribution is proven to be Gaussian (Equations 11 and 12 in Stephan et al. (2017)), they make additional assumptions such as the noise covariance matrix being independent of model parameters.

2. SGLD assumes a vanishing learning rate until convergence, whereas SGD is performed with a nonvanishing step size and for a finite number of iterations in practice. The learning rate of SGD is often large, which can cause model dynamics to drift even further from SGLD (Ziyin et al., 2023), especially in the discrete-time setting (Liu et al., 2021).

We thus characterize the analytical form of the posterior distribution with respect to model parameters trained with SGD:

Theorem 3.1 (Posterior for SGD). Assume the same assumptions as used in Theorems 2.2 and 2.3. Let w * be the local minimum that SGD (Equation 2) is converging towards. Then, the (conditional) log-probability of observing parameters w is given by (up to constants and negligible terms):

where L * = L(w * ), H * = H(w * ) and σ i (H * ) denotes the i-th largest eigenvalue of H * .

A proof for Theorem 3.1 is given in Appendix B. Theorem 3.1 suggests that the posterior distribution of model parameters learned by SGD not only relies on the training loss L(w) but is also crucially dependent on other terms, such as the Hessian structure H * , the gradient ∇L(w) and the ℓ 2 distance ∥w -w * ∥ 2 , confirming that Equation 5 is insufficient to model the dynamics of a discrete-time SGD algorithm.

Optimal Membership Inference under Discrete-time SGD

We have explained why the assumption imposed by Sablayrolles et al. (2019) about the posterior distribution of w following a Boltzmann distribution (Equation 5) does not hold for stochastic gradient methods typically employed in practice. Next, we prove a theorem that gives an estimate of the optimal membership inference scoring function by leveraging recent results on discrete-time SGD dynamics (Liu et al., 2021;Ziyin et al., 2021). Our derivation is based on the assumptions that the loss achieved at the local minimum is unaffected by removing a single training record and that the Hessian structure remains unchanged.

Assumption 3 (Similarity at local minimum). For any T and

. When the training dataset differs only by a single data point z 1 , assume that the Hessian matrix structure for models trained with and without the differing point share a similar structure, and the loss function also achieves a similar value at the local minimum:

where w * 0 is the local minimum that SGD with L 0 is converging towards, and H 0 denotes the Hessian matrix with respect to L 0 (and likewise for w * 1 , L 1 , and H 1 ).

As long as the size of the training dataset is sufficient and the excluded training record z 1 is not a lowprobability outlier from the data distribution D, we expect Assumption 3 generally holds for SGD algorithms.

Under Assumption 3 and a few other assumptions imposed in prior literature on discrete-time SGD dynamics (Liu et al., 2021;Ziyin et al., 2021), we obtain a theorem (proof is in Appendix C) that describes the scoring function for an optimal membership-inference adversary:

Theorem 3.2 (Optimal Membership-Inference Score). Given w produced by an SGD algorithm defined by Equation 2 and a record z 1 , the optimal membership inference M(w, z 1 ) is given by:

where I 1 , I 2 , I 3 , and I 4 are defined as follows:

Here, L * and H * are defined in Assumption 3, and are dependent on T . Here, T refers to the set of both member and non-member records along with their corresponding membership indicators, as defined in Lemma 2.1. Note that computing the optimal score requires access to the Hessian and model gradients, both of which require access to the model parameters. In fact, knowledge of the learning rate λ, momentum µ, and regularization parameter α are also required, thus requiring complete knowledge of the training setup of the target model. Thus, black-box access is not sufficient for optimal membership inference.

The first two additional terms I 1 and I 2 can be interpreted as the magnitude and direction, respectively, of a Newtonian step for the given record z 1 . The first term I 1 characterizes the influence magnitude in ℓ 2 -norm of upweighting z 1 on the model parameters close to the local minimum (Koh & Liang, 2017), while the second term captures the alignment between the influence of z 1 and the averaged influence of the remaining training data. A larger influence magnitude of z 1 or an increased influence alignment suggests a higher risk of membership inference. We remark that the notion of a self-influence function introduced in Cohen & Giryes (2024) naturally relates to I 1 , suggesting a similar insight to ours that better membership inference attacks can be designed by leveraging the influence function of the inferred record. The last two additional terms, I 3 and I 4 , originate from the extra L 2 regularization term imposed on the training loss of SGD (Section 2.2). When the regularization parameter α is a very small positive constant, the effects of I 3 and I 4 on optimal membership inference will be negligible, particularly compared to those of I 1 and I 2 .

Inverse Hessian Attack

While Theorem 3.2 directly prescribes an optimal membership inference adversary, computing the expectation over all possible models trained using the rest of the training data is infeasible. Our definition of optimal membership inference corresponds to the true leakage of the model (as defined in Section 3.2 of Ye et al. ( 2022)). It utilizes worst-case adversary knowledge (membership of all other training records) and white-box access to estimate the influence of the target record, similar to how empirical attacks such as LiRA (Carlini et al., 2022) and RMIA (Zarifzadeh et al., 2023) use reference models to account for atypical examples.

Specifically, making use of the insight of Theorem 3.2, we propose a scoring function based on the terms inside the expectation in Equation 7:

The score, IHA(z 1 ), for some given record z 1 , can be used as the probability of z 1 being a member. This serves directly as a useful attack for privacy auditing, without needing to train any reference models. Not having to train reference models offers significant advantages. It helps auditors avoid additional computational costs and, more importantly, eliminates the need for trainers to reserve hold-out data for reference model training.

This is particularly beneficial when data availability is a constraint for privacy auditing methods relying on reference models. While the absence of a negative sign with the loss function (like in LOSS) in IHA(z 1 ) may seem counter-intuitive at first glance, it can be rewritten such that it is proportional to the negation of the loss function (Appendix C.2).

While membership leakage is typically evaluated on a fixed dataset, the theoretical notion of optimal membership inference is defined for a much larger space. This space encompasses a broad distribution of possible data (including both member and non-member records) and the corresponding models trained on various splits of these datasets. In practice, it's challenging to realize this larger space, but if we could define true positive rates (TPRs) at low false positive rates (FPRs) with respect to this comprehensive space, they would empirically correspond to the optimal membership inference attack. This theoretical framework provides a more robust understanding of membership inference, though its practical implementation remains a significant challenge in privacy auditing.

The performance of our audit is also influenced by other factors, such as how efficiently and accurately the inverse-Hessian vector products (iHVPs) can be computed and to what degree our assumptions hold (particularly Assumption 3, which requires the Hessian and loss at local minima being unaffected by the exclusion of a single datapoint).

Experiments

To evaluate IHA, we efficiently pre-compute ∇L 1 (w) to facilitate the computation of ∇L 0 (w) for any given target record z 1 . For accurate Hessian matrix computations, we address the issue of ill-conditioning due to near-zero and small negative eigenvalues by either damping or using low-rank approximations (dampingbased conditioning seems to perform best; see Appendix E for details). To support larger models where direct Hessian computation is infeasible, we extend our method to use approximation methods based on Conjugate Gradients for iHVP computation (Koh & Liang, 2017). Our implementation for reproducing all the experiments is available as open-source code at https://github.com/iamgroot42/auditingmi.

Section 4.1 describes the baseline attacks, datasets, and models we use for our experiments. Section 4.2 summarizes our results, showing that IHA provides a robust privacy auditing baseline, matching or exceeding the performance of current state-of-the-art attacks including attacks that use reference models. This is notable since IHA does ont require training any reference models or the use of hold-out data.

For a given false positive rate (FPR), a threshold is computed using scores for non-members, which is then used to compute the corresponding true positive rate (TPR). This is repeated for multiple FPRs to generate the corresponding ROC curve, which is used to compute the AUC. This experimental design is commonly used for membership-inference evaluations (Yeom et al., 2018;Carlini et al., 2022;Ye et al., 2022).

Setup

To evaluate IHA, we compare its performance to state-of-the-art baseline attacks with a representative set of datasets and models.

Baseline Attacks. We include LOSS as a baseline that does not use reference models, SIF as it uses iHVP similar to our audit, and LiRA as it is the current state-of-the-art for membership inference. While RMIA (Zarifzadeh et al., 2023) uses fewer reference models, it achieves performance comparable to LiRA and thus for the sake of performance comparison, it suffices to use LiRA with a large number of reference models. We describe the underlying access assumptions for these attacks in Table 1.

LOSS (Yeom et al., 2018). The negative loss is used in this attack as a direct signal for membership inference.

SIF (Cohen & Giryes, 2024). Similar to ours, this attack employs the loss curvature of the target model by computing its Hessian, which is then used to compute a self-influence score. The original attack assigns 0-1 scores to target records. It classifies a given record as a member if its self-influence score is within the specified range and if its predicted class is correct. The latter rule can be ruled out as having many false positives/negatives. Instead of these steps, we choose to use the self-influence as membership scores directly.

While the authors used approximation methods for iHVP, we use the exact Hessian for fair comparison.

LiRA (Carlini et al., 2022). There are two variants, LiRA-Offline, which uses "offline" models to estimate a Gaussian distribution and then performs one-sided hypothesis testing using loss scores, and LiRA-Online, with additionally employs "online" models, i.e., models whose training data includes the target record. The likelihood ratio for online/offline model score distributions is then used as the score for membership inference.

We use LiRA-Online, since it is the strongest of the two variants.

L-Attack (Ye et al., 2022). The L-attack operates in a leave-one-out setting, training reference models on D \ {z} for any given record z. It uses loss as the target metric and computes attack thresholds for a desired false positive rate (FPR) by leveraging the distribution of losses obtained from reference models.

LiRA-L. We propose combining the LiRA attack for the LOO-setting by utilizing reference models trained under leave-one-out availability, followed by the offline variant of the LiRA attack (Carlini et al., 2022).

Datasets. Since we are limited by the computational constraints of computing iHVPs, we restrict our experiments to datasets where small models can perform adequately.

Purchase-100(S). The task for this dataset (Shokri et al., 2017) is to classify a given purchase into one of 100 categories, given 600 features. We train 2-layer MLPs (32 hidden neurons) with cross-entropy loss, with an average test accuracy of 84%. Experiments by Zarifzadeh et al. (2023) train larger (4-layer MLP) models on 25 K samples from Purchase-100, which is much smaller than the actual dataset, which is why we term it Purchase-100(S) (Small). We also demonstrate results with a 4-layer MLP that achieves similar task accuracy.

Purchase-100. For this version, we train models with 80 K samples. We use the same 2-layer MLP architecture as Purchase-100(S) but achieve a higher test accuracy of 90%. Using more data increases the scope for model performance. We report results for Purchase-100 in Table 2 as the corresponding models are less prone to overfitting. For completeness, we report results for Purchase-100(S) in Appendix D.

MNIST-Odd. We consider the MNIST dataset (LeCun et al., 1998), with the modified task of classifying a given digit image as odd or even. This modified task allows us to train models for binary classification using the regression loss, and is thus highly likely to follow the assumptions made in our theory regarding quadratic behavior for the loss function (Assumption 1). We train a logistic regression model with mean-squared error loss, with an average test loss of .078.

FashionMNIST. We use the FashionMNIST (Xiao et al., 2017) dataset, where the task is to classify a given clothing item image into one of ten categories. We train 2-layer MLPs (6 hidden neurons) with cross-entropy loss, with an average test accuracy of 83%. Models. We train 128 models in the same way as done in Carlini et al. (2022), where data from each model is sampled at random from the actual dataset with a 50% probability. For each target model and target record, there are thus 127 reference models available, half of which are expected to include the target record in the training data. All of our models are trained with momentum (µ = 0.9) and regularization (α = 5e -4 ), with a learning rate λ = 0.01. For a given false positive rate (FPR), a threshold is computed using scores for non-members, which is then used to compute the corresponding true positive rate (TPR). This is then repeated for multiple FPRs to generate the corresponding ROC curve, which is used to compute the AUC. This experimental design is commonly used for membership-inference evaluations (Yeom et al., 2018;Carlini et al., 2022;Ye et al., 2022).

Results

As summarized in Table 2, IHA provides a strong privacy auditing baseline that is competitive with current state-of-the-art attacks that require reference models. This is especially useful, considering that IHA does not require training any reference models and thus, does not require any hold-out data to train such reference models. IHA performs much better than the baselines on tabular data (Purchase-100), and is competitive with the baseline for image-based data (MNIST-Odd, Fashion MNIST). By extension, our method also outperforms previous membership inference attacks that specifically utilize parameter access via white-box access (Nasr et al., 2018;Cohen & Giryes, 2024), since such methods are outperformed by LiRA (Carlini et al., 2022). While tabular data is a more realistic setting for membership inference and the improved performance on Purchase-100 is promising, we leave to future work further investigation of these factors to better understand the performance discrepancies.

Approximating iHVPs. In order to carry out IHA, an auditor needs to be able to calculate iHVPs and gradients for all training data. While computing gradients is more computationally intensive than simply calculating the loss, the difference is minimal. On the other hand, computing an iHVP involves calculating the Hessian matrix and then inverting it, both of which are computationally expensive processes. Even storing such an inverted Hessian can be problematic (p × p matrix for a model with p parameters). We thus experiment with evaluating IHA using Conjugate Gradients (Koh & Liang, 2017) to approximate iHVP. While such approximation does not require computing the Hessian directly, the time taken to compute this term for each record is non-trivial. We thus evaluate this approximation-based method on a random sample of 10 000 recordsfoot_0 and find that approximation methods retain most of the attack's performance (Table 3).

We emphasize that the purpose of our comparisons is not to claim a better membership inference attack for adversarial use; the threat models are not comparable, since our attack requires knowledge of all other records D \ {z 1 } for inferring a given target record z 1 (relaxing this assumption leads to severe performance degradation, see Appendix F). Instead, IHA provides a way to empirically audit models for membership leakage without training reference models, which is desirable in avoiding the need to reserve hold-out data for training reference models. More importantly, our results suggest untapped opportunities in exploring parameter access for stronger privacy audits as well as the possibility of new white-box inference attacks from an adversarial lens.

Ablating over terms inside IHA

As described in Equation 8, calculating IHA requires computing the loss along with the four additional terms I 1 , I 2 , I 3 and I 4 . However, the terms I 3 and I 4 are scaled by α (which is usually very small) and involve an iHVP of an iHVP and thus may be much smaller compared to terms like I 1 , I 2 and the loss. We explore variants of IHA, which ignore the terms I 3 and I 4 , to see how they impact auditing performance. We also consider variants that use only the terms I 1 and I 2 to understand the importance of their contributions to the performance of IHA, along with the inclusion or not of the loss term to understand the relative importance of parameter-based signals.

The ablation study presented in Table 4 reveals several key insights about the performance of IHA. Excluding I 3 and I 4 has negligible impact on auditing performance, even for low-FPR scenarios. The combination of I 1 and I 2 alone achieves an AUC of .779, and performance is identical when the terms I 3 and I 4 are also included.

Notably, the addition of the loss term ℓ(w, z 1 ) to I 1 and I 2 results in a marginal improvement, increasing the AUC to .791 and slightly boosting the TPR at both 1% and 0.1% FPR. Interestingly, when examined individually, I 2 (AUC .704) performs significantly better than I 1 (AUC .591), suggesting that I 2 captures more relevant information for the auditing task. Including the loss term ℓ(w, z 1 ) has little impact on I 1 but harms performance when used wth I 2 . These findings indicate that the majority of the attack's effectiveness stems from the inverse Hessian vector products used in I 1 and I 2 , with I 2 being particularly important, while the terms involving weight regularization and nested iHVPs (I 3 and I 4 ) contribute minimally to the overall performance. While not as impactful as I 2 , the loss term still provides valuable information for the auditing process. Based on these results, a simplified version of IHA using only ℓ(w, z 1 ), I 1 , and I 2 could potentially offer a favorable trade-off between computational efficiency and auditing effectiveness.

Comparison with Leave-One-Out Setting

When targeting a record for inference, IHA assumes knowledge of all the other n -1 records in an n-sized dataset. It is possible that the improved performance of IHA is due to this extra information rather than inherent parameter access. To isolate and analyze these potential sources of increase in leakage, we also assess the performance of a leave-one-out (LOO) membership inference test. We evaluate the L-attack (Ye et al., 2022) on 1000 samples, training 100 reference models per record for score calibration. Since targeting each record requires training multiple reference models for the L-attack, evaluating it on a larger sample of data is computationally infeasible.

These results indicate that IHA outperforms the L-attack, achieving an AUC value of .791 compared to .737 for the L-attack.foot_1 This suggests that even when controlling for the additional knowledge of all other records in the dataset, the primary source of IHA's superior performance stems from its access to model parameters rather than just the leave-one-out setup. In a way, IHA uses parameter access to obviate the need for reference models, as it directly aims to measure the influence of the given target record instead of relying on reference models for score calibration.

Interestingly, in comparison, LiRA achieves an AUC of .767, which is lower than IHA but still higher than the L-attack. This suggests that LiRA, even without the extensive reference model training, is more effective than the L-attack, possibly due to its utilization of both "in" and "out" models as opposed to just "out" models with the L-attack. To try and devise a stronger black-box attack for the LOO setting, we extend LiRA to the LOO setting by using models trained on LOO data as reference models. LiRA-Offline under the LOO setting achieves an AUC of .633, lower than the L-attack (.737). Overall, these results demonstrate the promise of IHA as a privacy auditing tool. It yields results comparable to that of techniques that train hundreds of reference models (for each target record in the worst case, as in L-attack), without using a single reference model.

Inter-Attack Agreement

Similar to Ye et al. (2022), we compute the agreement rate between ground-truth membership labels and membership predicted by various attacks to understand the ability of our privacy audit to identify vulnerable data, and demonstrate how it differs from existing attacks. Table 5 presents the agreement rate between ground-truth membership labels and membership predicted by various attacks.

At a low FPR of ≈ 0.05, agreement in predictions for non-members is very high between attacks, with agreement rates above 0.91 for all pairs of attacks. On the other hand, agreement rates for member records are expectedly lower. Interestingly, agreement between LiRA and LiRA-L is higher than IHA and any other attack. This difference is especially pronounced for a higher FPR (Table 5b), where agreement rates are as low as ≈ 0.4 compared to 0.766 for LiRA and LiRA-L. This is very interesting becaue the corresponding TPRs for IHA are higher than LiRA and comparable to that of the L-attack, thus suggesting that the records identified by IHA as being vulnerable are very different from those identified by LiRA or even the L-attack. This also means that a combined (classifying a record as a member only when both attacks classify as a member) attack would have some true positives with a very low FPR.

Runtime Comparison

To compare the computational costs of our proposed audit with existing auditing techniques, we analyze runtime and memory usage statistics across different methods, aiming to evaluate efficiency and practicality in real-world privacy audits (Table 6).

While the peak memory consumption of IHA is higher than that of LiRA, the approximate version of IHA is not too far from LiRA in terms of memory consumption. Computing the total runtime is a function of the number of samples used for the privacy audit, as a "precompute" step is required for LiRA (training reference models) and exact IHA (computing Hessian). For instance, computing the audit for 1K samples takes less overall time for IHA (≈ 1 hour) than it does for LiRA (≈ 3 hours). It should be noted that the Hessian is too large to store on our GPU for IHA and is thus stored on the CPU, which is also why it is slower. Although improvements may reduce the compute costs, the key advantage of such a privacy audit comes from not having to reserve hold-out (or auxiliary) data. Our privacy audit, like the most trivial LOSS attack, only requires member data and some non-member data, whereas other attacks in the literature require shadow/reference models trained on comparable-sized datasets; the limiting factor here is reserving data to train reference models, which a real-world model trainer may not want to do since it reduces the amount of data available for training.

Conclusion

Our theoretical result proves that model parameter access is indeed necessary for optimal membership inference, contrary to previous results derived under unrealistic assumptions and the common belief that optimal membership inference can be achieved with only black-box model access. We propose the Inverse Hessian Attack inspired by this theory that provides stronger privacy auditing than existing black-box techniques.

Limitations. IHA is not yet practically realizable for most settings due to the computational expense of calculating the Hessian, or even approximating iHVPs. This restriction poses challenges for real-world cases where fixed compute budgets may be more crucial than the availability of auxiliary data. An auditor might use a subset of parameters to reduce computational costs while performing Hessian-based computations. This aligns with model pruning (Liu et al., 2019), but understanding its impact on membership knowledge within parameters is non-trivial (Yuan & Zhang, 2022). We also note that IHA's performance can be sensitive to the choice of the damping factor, which requires further investigation (Appendix D.1).

Our conclusion aligns well with recent calls in the literature to consider white-box access for rigorous auditing (Casper et al., 2024). While our theory shows that parameter access is required for optimal membership inference, it remains unclear how much better this is compared to the optimal membership inference attack restricted to black-box access. Our empirical studies suggest the gap is non-trivial, but further study is required to understand the theoretical limit of black-box attacks, which is a non-trivial but interesting direction to explore. Exploring the accuracy of iHVP approximation methods to extend IHA to larger models, along with multi-record inference, are both promising directions for future research.

Broader Impact Statement

The increasing integration of AI in sensitive domains like healthcare, finance, and personal data management highlights the critical importance of privacy. Information leakage from AI models can have severe consequences, making effective privacy auditing a necessary safeguard. Our work contributes to this field by theoretically demonstrating that optimal membership inference attacks require white-box access to model parameters, challenging the adequacy of black-box approaches. We also demonstrate with Inverse Hessian Attack how this theory can be used to design empirical privacy audits that do not rely on reference models.

We advocate for the development of more sophisticated privacy auditing tools that fully leverage the elevated access typically available to auditors, such as model parameters, to assess privacy leakage efficiently without extensive data and compute resources. We hope our theoretical and empirical results will reinvigorate interest in the privacy research community to explore white-box attacks, for both adversarial and auditing purposes.

White-box

Membership Inference. Nasr et al. (2018) explored white-box access to devise a metaclassifier-based attack that additionally extracts intermediate model activations and gradients to increase leakage but concluded that layers closer to the model's output are more informative for membership inference and report performance not significantly better than a black-box loss-based attack. Recent work by DeAlcala et al. ( 2024), however, makes the opposite observation, with layers closer to the model's input providing noticeably better performance. Apart from these meta-classifier driven approaches, some works attempt to utilize parameter access much more directly, often utilizing Hessian in one form or another. Cohen & Giryes (2024) defined the self-influence of a datapoint z i as (g i ⊤ H -1 g i ) as a signal for membership, using LiSSA (Agarwal et al., 2017) to approximate the iHVP. This has similarities to our result since our optimal membership inference score also involves computing iHVPs. Li et al. (2023) attempted to measure the sharpness for a given model by evaluating fluctuations in model predictions after adding zero-mean noise to the parameters, a step that is supposed to approximate the trace of the Hessian at the given point.

A.2 Privacy Auditing

Ye et al. ( 2024) proposed using efficient methods to "predict" memorization by not having to run computationally expensive membership inference attacks, with reported speedups of up to 140x. They showed how their proposed score (LOOD) correlates well with AUC, corresponding to an extremely strong MIA with all-but-one access to records (L-attack (Ye et al., 2022)). However, it is unclear if this computed LOOD is directly comparable across models, making it hard to calibrate these scores to compare the leakage from a model relative to another (an important aspect of internal privacy auditing). Their derivations also involve a connection with the Hessian. Biderman et al. (2024) studied the problem of forecasting memorization in a model for specific training data and proposed using partially trained versions of the model (or smaller models) as a proxy for their computation. While their results support the need for inexpensive auditing methods, their focus is on predicting memorization early in the training process, while ours relates to auditing fully trained models. More recently, Tan et al. (2022) studied the theory behind worst-case membership leakage for the case of linear regression on Gaussian data and derived insights. While this is useful to make an intuitive connection with overfitting, it does not provide a realizable attack or insights for the standard case of models trained with SGD.

A.3 SGD Dynamics and iHVPs

SGD Dynamics. Stephan et al. (2017) approximated the SGD dynamics as an Ornstein-Uhlenbeck process, while Yokoi & Sato (2019) provided a discrete-time weak-order approximation for SGD based on Itô process and finite moment assumption. However, both works rely on strong assumptions about the gradient noises and require a vanishingly small learning rate, largely deviating from the common practice of SGD. To address the limitations of the aforementioned works, Liu et al. (2021) directly analyzed the discrete-time dynamics of SGD and derived the analytic form of the asymptotic model fluctuation with respect to the asymptotic gradient noise covariance and the Hessian matrix. Ziyin et al. (2021) further generalized the results of (Liu et al., 2021) by deriving the exact minibatch noise covariance for discrete-time SGD, which is shown to vary across different kinds of local minima. Our work builds on these advanced theoretical results of discrete-time SGD dynamics but aims to enhance the understanding of optimal membership inference, particularly for models trained with SGD.

iHVPs. Currently literature on approximating inverse-Hessian vector products relies on one of two methods: conjugate gradients (Koh & Liang, 2017) or LiSSA (Agarwal et al., 2017). Both approximation methods rely on efficient computation of exact Hessian-vector products, and use forward and backward propagation as sub-routines. While these methods have utility in certain areas, such as influence functions (Koh & Liang, 2017) and optimization (Oldewage et al., 2024), approximation errors can be non-trivial. For instance, I 1 in the formulation of our attack requires a low approximation error in the norm of an iHVP, while I 2 simultaneously requires a low approximation error in the direction of the iHVP. Recent work on curvatureaware minimization by Oldewage et al. (2024) proposes another method for efficient iHVP approximation as a subroutine, but the authors observed high approximation errors based on both norm and direction.

B Proof for Theorem 3.1

Proof. Recall that H * = H(w * ) and L * = L(w * ). According to Theorem 2.2 and Theorem 2.3, we obtain

where we only consider the leading terms in Theorems 2.2 and 2.3. Note that Equation 9 holds when the Hessian matrix H * has full rank and L * ̸ = 0. When the Hessian has degenerated rank such that rank(H * + αI d ) = r < d, the following more generalized result can be derived:

, where P r = diag(1, ..., 1, 0, ...0) denotes the projection matrix onto non-zero eigenvalues, and + is the Moore-Penrose inverse operator. If L * = 0, meaning w * is a global minimum, then the asymptotic model fluctuation Σ = 0. For ease of presentation, we assume the Hessian matrix has full rank in the following proof.

Then, we get:

Using the Sherman-Morrison formula, we obtain:

Leaving out the second-order term O(α 2 ) in Equation 11(since the regularization parameter α is a typically small constant in [0, 1)) and plugging it back in Equation 10, we get:

According to Laplace approximation, we can approximate the posterior distribution of w given w * as N (w * , Σ). Therefore, making use of Equation 9, we can derive the explicit formula of the log-posterior distribution as:

Note that we assume the model parameterized by w is converging towards some local minima w * using stochastic gradient descent, where the loss landscape around the local minima w * has a quadratic structure.

Therefore, we can perform second-order Taylor expansion of the loss function at w * as follow:

which further suggests that ∇L(w) = H * (ww * ) if taking gradient with respect to w for both sides of Equation 14. Plugging Equation 14 into Equation 13, we further obtain:

Omitting the constant and negligible terms, we thus complete the proof of Theorem 3.1. Note that we keep the O(α 2 ) term in Equation 13 and Theorem 3.1 for the sake of completeness but expect it to be negligible compared with other terms, due to the fact that α is typically set as a very small constant within [0, 1).

C Optimal Membership-Inference Score

C.1 Proof for Theorem 3.2

Proof. To derive the scoring function for an optimal membership inference, we need to compute the ratio between p(w|w * 1 ) and p(w|w * 0 ), where w * 0 (resp. w * 1 ) denotes a local minimum (close to w) of the training loss function with respect to {z 2 , . . . , z n } (resp. {z 1 , . . . , z n }). Note that we've obtained the posterior distribution of w in Theorem 3.1. Therefore, the remaining task is to analyze the following terms:

where H 0 (w * 0 ) (resp. H 1 (w * 1 )) denotes the Hessian of L 0 (resp. L 1 ) at w * 0 (resp. w * 1 ). Since both w * 0 and w * 1 are close to parameters of the observed victim model w, so we can approximate the corresponding loss using second-order Taylor expansion. Also, according to Assumption 3, we know H 0 (w

Thus, we can simplify Equation 16 and obtain the following form: ln p(w|w * 1 ) -ln p(w|w * 0

where the second equality holds because of the Taylor approximation ∇L i (w) -∇L i (w * i ) = H * (ww * i ) for i ∈ {0, 1}. Moreover, according to Lemma 2.1, we know the optimal membership inference is given by:

where σ(u) = (1 + exp(-u)) -1 is the Sigmoid function, T = {z 2 , . . . , z n , m 2 , . . . , m n }, and γ = P(m i = 1).

Plugging Equation 17into Equation 18, we obtain

where I 1 , I 2 , I 3 , I 4 and t γ are defined as:

Therefore, we complete the proof of Theorem 3.2.

C.2 Connection with LOSS attack

Note that while there are additional terms in our optimal membership-inference score, there is another critical difference: the loss function has its sign flipped when compared to existing results (Yeom et al., 2018;Sablayrolles et al., 2019). While this may seem counter-intuitive at first glance, we show below the addition -(I 1 + I 2 + I 3 + I 4 ) terms in Equation 7 are expected to be negatively correlated to the loss function, leading to the proposed scoring function, in fact, aligns with the intuition of existing results. For simplicity, we consider the setting without regularization (i.e., α = 0) and the Hessian matrix has full rank.

According to the assumption of quadratic loss around w * , we have the following observations:

where U ⊤ diag{σ 1 , . . . , σ d }U is the eigenvalue decomposition of H. Let v = U(ww * ). Since U is an orthonormal matrix, we know ∥v∥ 2 = ∥w -w * ∥ 2 . Thus, we obtain

which further suggests (provided the Hessian has full rank)

Based on Assumption 3, we assume that the Hessian structure and the loss function value remain unchanged with and without a single record z 1 . We hypothesize that the ratio

Note that the derivation from Equation 19to Equation 20 is not mathematically rigorous, but as long as the record z 1 is not too deviated from the data distribution, we expect the above inequality holds. Plugging Equation 20 into the log-likelihood term inside M(w, z 1 ) (first equality in Equation 17with α = 0), we get ln p(w|w * 1 ) -ln p(w|w * 0 ) =

where k is some real number that falls into [σ d , σ 1 ]. In addition, for the case of full-rank Hessian, it is easy to see that the i-th eigenvalue of Σ (Equation ( 4)) can be written as:

Since the covariance matrix Σ is positive semi-definite and invertible, it must follow that all of its eigenvalues are positive:

2 -λσ i 1 + µ > 0 ⇒ 1 1 + µ -2 λσ i < 0, for any i = 1, 2, . . . , d.

With the above inequalities in mind, by looking at Equation ( 21), we get: ln p(w|w * 1 ) -ln p(w|w * 0 ) =

The upper limit on the score (hence the score itself) corresponding to IHA is thus proportional to the negative of the loss function, aligning with intuition (lower loss indicative of overfitting, and thus the record being a member). The score inside IHA can thus be interpreted (up to some approximation error) as -f (w, z 1 )ℓ(w, z 1 ) for some f (w, z 1 ) > 0 that essentially accounts for SGD training dynamics, and is a function dependent on parameter access to the target model.

Dataset

Low-Rank Damping ϵ = 1e -2 ϵ = 1e -1 ϵ = 2e -1 ϵ = 1e -2 ϵ = 1e -1 ϵ = 2e

-1 MNIST-Odd .521 .530 .500 .513 .535 .542 FashionMNIST .551 .557 .541 .533 .582 .594 F Approximating L 0

For auditing purposes, experiments where all but one member is known are useful, but an adversary is unlikely to have this much knowledge of the training data. We experiment with the potential use of IHA where only partial knowledge of the remaining n -1 members may be available to approximate ∇L 0 , Approximating L 0 with a fraction of the actual dataset could be useful in not only reducing the computational cost of the audit, but also potentially enabling adversarial use of the attack in threat models where the attacker has partial knowledge of the training data. We evaluate IHA for versions where L 0 is approximated using a randomly-sampled fraction of the training data and report results in Table 9. We see a clear degradation in performance when only a subset of data is used-this is especially true for lower fractions, where AUC can drop by as much as ≈ 0.2. More importantly, even when using 90% of the training data, there is a significant gap in performance. The statistics we compute for IHA thus do completely utilize knowledge of all other training records. While this result suggests that adversarial use of IHA would require a very strong adversary (that posseses knowledge of nearly all training records), it also hints at how data poisoning attacks could have a large impact on downstream membership inference. A poisoning adversary could hypothetically craft data a way that interferes with L 0 (when relating to the optimal membership adversary) and thus increase/decrease inference risk for other records.