<?xml-model href='http://www.tei-c.org/release/xml/tei/custom/schema/relaxng/tei_all.rng' schematypens='http://relaxng.org/ns/structure/1.0'?><TEI xmlns="http://www.tei-c.org/ns/1.0">
	<teiHeader>
		<fileDesc>
			<titleStmt><title level='a'>Do You Really Need Public Data? Surrogate Public Data for Differential Privacy on Tabular Data</title></titleStmt>
			<publicationStmt>
				<publisher>Neural Information Processing Systems Foundation, Inc. (NeurIPS)</publisher>
				<date>01/01/2025</date>
			</publicationStmt>
			<sourceDesc>
				<bibl> 
					<idno type="par_id">10710257</idno>
					<idno type="doi">10.52202/085713-0340</idno>
					
					<author>Shlomi Hod</author><author>Lucas Rosenblatt</author><author>Julia Stoyanovich</author>
				</bibl>
			</sourceDesc>
		</fileDesc>
		<profileDesc>
			<abstract><ab><![CDATA[Differentially private (DP) machine learning often relies on the availability of public data for tasks like privacy-utility trade-off estimation, hyperparameter tuning, and pretraining. While public data assumptions may be reasonable in text and image data, they are less likely to hold for tabular data due to tabular data heterogeneity across domains. We propose leveraging powerful priors to address this limitation; specifically, we synthesize realistic tabular data directly from schema-level specifications — such as variable names, types, and permissible ranges — without ever accessing sensitive records. To that end, this work introduces the notion of "surrogate" public data — datasets generated independently of sensitive data, which consume no privacy loss budget and are constructed solely from publicly available schema or metadata. Surrogate public data are intended to encode plausible statistical assumptions (informed by publicly available information) into a dataset with many downstream uses in private mechanisms. We automate the process of generating surrogate public data with large language models (LLMs); in particular, we propose two methods: direct record generation as CSV files, and automated structural causal model (SCM) construction for sampling records. Through extensive experiments, we demonstrate that surrogate public tabular data can effectively replace traditional public data when pretraining differentially private tabular classifiers. To a lesser extent, surrogate public data are also useful for hyperparameter tuning of DP synthetic data generators, and for estimating the privacy-utility tradeoff.]]></ab></abstract>
		</profileDesc>
	</teiHeader>
	<text><body xmlns="http://www.tei-c.org/ns/1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xlink="http://www.w3.org/1999/xlink">
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1">Introduction</head><p>Differential privacy (DP) is a mathematical framework for protecting individuals' privacy in statistical analysis and machine learning <ref type="bibr">[38]</ref>, and was deployed in multiple recent high-stakes releases and systems <ref type="bibr">[3,</ref><ref type="bibr">55,</ref><ref type="bibr">83,</ref><ref type="bibr">21,</ref><ref type="bibr">113,</ref><ref type="bibr">40]</ref> (see <ref type="bibr">[35]</ref> for a more complete list). It is common in the design of differentially private algorithms to assume access to a relevant public dataset that can guide hyperparameter tuning, pretraining, or performance improving mechanisms <ref type="bibr">[11,</ref><ref type="bibr">13,</ref><ref type="bibr">71,</ref><ref type="bibr">123]</ref>. Executing these tasks with sensitive data would require an additional allocation of the privacy budget, resulting in weaker overall privacy guarantees or reduced utility. However, using this assumed public data in a private mechanism avoids additional privacy budget consumption. This leads to the following informal definition of public data in our work:</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Public Data (informal)</head><p>A dataset is considered public if a computation taking it as input does not consume privacy loss budget with respect to any fixed private, sensitive dataset.</p><p>For text and image domains, assuming public data availability is often reasonable: public image collections or large-scale textual corpora are readily available, and it has been shown that even outof-distribution data can serve as a valuable prior in these contexts, whether through pretraining or foundation models <ref type="bibr">[85,</ref><ref type="bibr">42]</ref>. However, this assumption does not often hold in a tabular data setting. Tabular data is heterogeneous, high-dimensional, subject to strict privacy or legal restrictions, and has few universal priors <ref type="bibr">[84]</ref>. In many real-world domains like healthcare, finance, and government administration, tabular data encodes sensitive information that drives high-stakes decisions. It is thus rare to find truly public, non-sensitive samples with sufficient alignment to a private distribution to be used for private hyperparameter tuning or pretraining.</p><p>Figure <ref type="figure">1:</ref> (Top) The premise of this work: Can we utilize LLMs to generate surrogate public data to solve DP auxiliary tasks? (Bottom) The answer is yes; for example, pretraining on surrogate public data generated through various LLM based methods (green, orange) nearly matches the performance of pretraining on regular public data ( magenta), and outperforms no pretraining (red), or pretraining on baselines (gray, blue). Results on the EDAD dataset.</p><p>Nevertheless, recent theoretical insights confirm that if a public dataset is "close enough" to a sensitive data distribution, then private learning can still achieve strong utility, even when the public and private datasets are not perfectly matched <ref type="bibr">[11]</ref>. In practice, however, identifying or constructing such a surrogate is often far from trivial. Real-world deployments of differentially private methods face numerous hurdles related to data availability <ref type="bibr">[30,</ref><ref type="bibr">31]</ref>. As an example, a recent release of Israel's Live Birth Registry <ref type="bibr">[55]</ref> underscores the challenges of obtaining an end-to-end differential privacy guarantee.</p><p>Public data served two purposes for Hod and Canetti <ref type="bibr">[55]</ref>: it helped constrain the hyperparameter space within a computationally lockeddown enclave environment, and it enabled the estimation of the privacy-utility trade-off when allocating privacy budget. Yet, in general, sensitive datasets (e.g., birth records) are not readily available as public data. Hod and Canetti <ref type="bibr">[55]</ref> reported finding only one open-access birth dataset worldwide (in the U.S.); without it, estimating the necessary parameter settings for their release would have been significantly more challenging.</p><p>A recent practical guide for differentially private machine learning recommends that "the simplest approach, when possible, is to do all model architecture search and hyperparameter tuning on a proxy public dataset (with a distribution similar to the private data), and only use the private training dataset to train the final DP model" <ref type="bibr">[89]</ref>.</p><p>These two examples highlight a fundamental challenge; many differentially private algorithms require informed decisions a priori that, ideally, do not consume extra privacy budget. This leads us to consider a class of DP auxiliary tasks, which we define informally as: Differential Privacy Auxiliary Task A differential privacy auxiliary task, with respect to a differentially private mechanism for conducting an analysis of interest, is a required decision or procedure for execution. The auxiliary task may or may not incur privacy loss. Examples include hyperparameter tuning, setting &#949;, mechanism initialization, model selection, etc. requires a careful design across the DP downstream task, datasets, baselines, comparison conditions, and aggregated metrics. In this work, we propose such a benchmark framework and provide an extensible, method-agnostic implementation.</p><p>(3.) In-depth experimental results evaluating surrogate public data on some DP auxiliary tasks (Section 5). We find that pretraining with LLM-generated surrogate public data can substantially improve differentially private classification performance; this holds true in the low dataset size regime in particular. Additionally, we show that LLM-generated surrogate public data can be useful for hyperparameter tuning of private data synthesizers. We further present a complicated story on using surrogate public data for privacy-utility tradeoff estimation (i.e. "setting the privacy budget").</p><p>The code used to generate the surrogate public data and execute all experiments is publicly available. <ref type="foot">2</ref> Additionally, we emphasize that the approaches to surrogate public data we will discuss are a fallback for settings lacking a matched traditional public sample. If a sufficiently similar traditional public dataset exists, it should generally be preferred, as it also incurs no privacy cost under Def. 2.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2">Preliminaries</head><p>Differential privacy (DP) ensures that the presence or absence of a single individuals data has only a limited influence on an output statistic; in other words, it restricts how much any single record can affect the outcome of an analysis. To define this, we consider two datasets D, D &#8242; &#8712; X n , which are neighboring if they differ in at most one data entry. Let X denote the universe of records. Definition 1 (Differential Privacy <ref type="bibr">[38]</ref>). An algorithm M : X n &#8594; R satisfies (&#949;, &#948;)-differentially private if, for every pair of neighboring datasets D, D &#8242; &#8712; X n , and for every subset of possible outputs S &#8838; R,</p><p>The following definition of public data is inspired by <ref type="bibr">[14]</ref>. Definition 2 (Public Data). A dataset D &#8712; X m is public if incorporating it into any computation does not incur additional privacy loss. That is, for any sensitive dataset D &#8712; X n and for every (&#949;, &#948;)-differentially private mechanism M, the privacy guarantee is identical whether D is used or not, i.e., M(D, &#8226;) and M(D, D) both satisfy identical (&#949;, &#948;)-differential privacy guarantees.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3">Producing Public Data Surrogates</head><p>We evaluate multiple methods for generating surrogates to public data, categorizing them into baseline and LLM-based approaches. For these methods, we assume that the private data's metadataconsisting of the dataset schema and a brief description of its topic (e.g., demographics, epidemiology) -is publicly available. All methods we introduce rely solely on this metadata. <ref type="foot">3</ref> A schema provides a description of the dataset domain and structure, specifying for each variable: (1) its name, (2) a very brief description, (3) the data type (e.g., integer, string), and (4) either allowed values and their meanings for categorical columns or value ranges for continuous columns. This metadata is typically extracted from the dataset's accompanying README file or codebook (see, e.g., <ref type="bibr">[69]</ref> on ICPSR). Figure <ref type="figure">5</ref> is an excerpt from a schema. Each LLM-based method is applied to the three models presented in Table <ref type="table">2</ref>. Below we briefly summarize each approach; see Appendix C for full details, and Figure <ref type="figure">1</ref> for an overview.</p><p>Baselines. We evaluate three baseline methods: Uniform and Arbitrary rely solely on the public schema; Univariate intentionally uses noisy one-way marginals from private data as a nonsurrogate comparison point. The Uniform method samples records i.i.d. from each variables full domain, while the Univariate approach samples columns independently using empirical 1way marginal distributions from the private data (this violates privacy, but serves as a competitive baseline). The Arbitrary method constructs a random Bayesian network (BN) over the highdimensional domain by sampling a DAG (maximum in-degree of 5) and parameterizing each nodes conditional probability tables from symmetric Dirichlet priors; see Algorithm 1. CSV (direct generation). The CSV method prompts LLMs to generate CSV records that strictly adhere to the schema, including exact header rows, data types, allowed values, and realistic interfield relationships (see Figure <ref type="figure">6</ref>). The prompt specifies rules to ensure statistical plausibility and the inclusion of realistic edge cases, while the generation is executed in batches with schema-based validation of each record. This process relies solely on the LLMs pretrained knowledge without any direct access to the private data. When the schema specifies numeric ranges (e.g., float in [a, b]), the prompt and validator accept continuous values; in practice Claude/GPT emit decimals within range that we can naturally retain when downstream mechanisms support continuous features.</p><p>Agent (state machine) approach. The Agent method (implemented as a state machine, see Figure <ref type="figure">7</ref>) is a multi-step process to construct a structural causal model (SCM) from the schema metadata. It begins by describing the full set of variables and domain-specific constraints, then sequentially constructs a causal DAG -first identifying root nodes and then establishing edges (ensuring acyclicity deterministically) before mapping variables to structural equations. For numeric nodes, the Agent selects a parametric family (e.g., Gaussian, Log-Normal, Gamma) and emits full Pyro sampling code, yielding continuous values by construction. We validate acyclicity and node coverage with NetworkX before code generation. Then, the final output is an integrated Pyro Python program that enforces variable ranges and constraints. The Agent method has two variants: we experiment with generating multiple expert models whose records are re-sampled (using uniform or facility location-based sampling <ref type="bibr">[111]</ref>).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4">Evaluation Framework</head><p>Our evaluation framework assesses the viability of the surrogate public data in three DP auxiliary tasks: (1) classifier pretraining, (2) hyperparameter optimization, and (3) privacy-utility trade-off estimation. Each task is assessed using three datasets, and corresponding DP mechanisms. Our strategy in evaluating each task is guided by a high level question: how useful is each surrogate public data method relative to traditional public data and relative to the lack of any public data? Figure <ref type="figure">2</ref>: An overview of our evaluation framework (Section 4) We assess the usefulness of regular public data and surrogate public data (Section 3) on three tasks. Two tasks are related to synthetic data generation -hyperparameter tuning and privacy-utility estimation -and one to classification model pretraining.</p><p>These three auxiliary tasks (public pretraining, hyper-parameter tuning, and privacy-utility estimation) are repeatedly emphasized as high-impact components in deployed DP pipelines <ref type="bibr">[89]</ref>. We therefore concentrate our benchmark on them and report ( , &#948;) and other settings inline (full grids in Appendix D.4). We report standard error over 10 seeds where appropriate.</p><p>Task 1: model pretraining for classification. We assess the benefit of surrogate public data as a pretraining step for binary classification on tabular data using an FTTransformer model <ref type="bibr">[46,</ref><ref type="bibr">95]</ref>. Public and private datasets are split into train, validation, and test sets <ref type="bibr">(72:8:20)</ref>, and performance is measured via AUC along with an AUC Advantage metric comparing models with and without public pretraining, aggregated over multiple experiment seeds. See Appendix D.1.1 for the detailed specification of this task.</p><p>Task 2: hyperparameter tuning for synthetic data generation. We evaluate whether surrogate public datasets can effectively guide hyperparameter selection for DP synthetic data generators across varying privacy budgets &#949;. For each synthesizer and hyperparameter configuration, we generate synthetic datasets matching the size of the private data and measure their performance on the private data using multiple metrics (marginals, correlations, and classification-based; see Table <ref type="table">3</ref>). We quantify performance degradation by comparing results obtained when tuning hyperparameters on surrogate public data versus directly on private data (optimal choice of hyperparameters), aggregating outcomes across multiple experimental seeds via Pareto frontier analysis. See Appendix D.1.2 for the detailed specification of this task.</p><p>Task 3: privacy-utility estimation for synthetic data generation. Finally, we assess how accurately surrogate public datasets can approximate the privacy-utility trade-off curve of a DP synthetic data mechanism fitted to private data. For each combination of synthesizer, dataset, and metric group, we select the best-performing hyperparameters based on surrogate public datasets, and then evaluate the DP mechanism across a range of &#949; values. This produces paired performance curves: one based on the private data (true curve) and others based on surrogate public datasets. We quantify the dissimilarity between these curves using the &#8467; 1 and &#8467; 2 distances. See Appendix D.1.3 for the detailed specification of this task.</p><p>Datasets. We run the experiments on three datasets (ACS, EDAD, and WE; high-level details presented in Table <ref type="table">1</ref>). Each dataset has a private, sensitive split; additionally, we pair each dataset with a reasonable public analogue. These public datasets have inherent distribution shift between them; for ACS this is a geographical variation (assuming the Massachusetts sample is publicly available, while a more diverse national sample is private) or, for EDAD and WE, temporal differences (versions of the same survey from prior years). All datasets contain only categorical features to ensure compatibility with synthetic data generation methods. The private split serves as ground truth to benchmark the contribution of the "traditional" approach of using a public split compared to our surrogate generation methods. To mitigate the risk of data memorization in LLMs, we specifically selected the private splits for EDAD and WE to be recently published, i.e., after the training data cutoff of some of the LLMs we evaluate. To this end, we include a memorization analysis in Appendix D.2.4, based on the methodology of Bordt et al. <ref type="bibr">[18]</ref>. Concretely, EDAD and WE private splits were publicly released on April 30, 2024, after GPT-4o (Oct 2023) and Llama-3.3 (Dec 2023) cutoffs, and effectively aligned with Claude 3.5 Sonnets "up until April 2024" window. <ref type="foot">4</ref> For the complete details for each dataset and an in-depth discussion of LLM memorization, refer to Appendix D.2.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Mechanisms.</head><p>Our private mechanisms encompass differentially private classification (Task 1) and data synthesis (Tasks 2 &amp; 3). As discussed previously, for classification, we employ an FTTransformer model <ref type="bibr">[46]</ref> -a transformer-based architecture tailored for tabular data that rivals gradient boosting methods like XGBoost -by adapting it with minor modifications to support DP-SGD for private fine-tuning and allowing pretraining with public data via standard gradient updates <ref type="bibr">[1,</ref><ref type="bibr">95]</ref>. For private data synthesis, we evaluate three state-of-the-art methods -PrivBayes, GEM, and AIMthat follow the "Select-Measure-Project" paradigm: they privately select statistical queries (e.g., kway marginals or correlations) on sensitive data, add noise to these measurements, and then project the results onto a synthetic distribution <ref type="bibr">[120,</ref><ref type="bibr">73,</ref><ref type="bibr">80]</ref>. See Appendix D.  Table 3: Pareto Efficient Methods for PrivBayes on WE. Figure 3: Task 1 -Pretraining: (a, b) Comparing the mean AUC on the test subset private split for the Pretraining model vs. the Fine-tuned model, grouped by generation method (mean calculated across DP finetuning parameter space when the best configuration is chosen with the validation subset of the public split for the pretraining step, across 10 runs). Note how the starting point of model AUC differs, while the improvement from private finetuning (i.e. the increase in AUC) is relatively stable. Task 2 -Hyperparameter tuning: (Tables 1, 2 and 3) show some Pareto frontiers for the performance degradation metric when hyperparameter tuning using (surrogate) public data methods for tuning relative to tuning on private data. Note how CSV and Agent methods are competitive with tuning on the regular public data. See Section 5 and Appendix E for complete results. Note that we adopt the Olympic medal convention in each table in our paper: gold , silver and bronze cells signify first, second and third best performance, respectively.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5">Results</head><p>Task 1: Model Pretraining for Classification. Our experiments provide strong evidence that LLM-based methods -both CSV and Agent generation (notably with Claude 3.5 Sonnet) -offer a competitive alternative to traditional public data in the small dataset regime (fewer than 10K records).</p><p>In many settings, the surrogate public data matched (or even occasionally exceeded) the performance of regular public data as a starting point for DP fine-tuning. Figure <ref type="figure">3</ref> presents our experimental results on the EDAD and WE datasets, demonstrating how pretraining on the surrogate public data can vastly improve the starting point of model performance. For the ACS dataset, we do not observe a benefit from pretraining, either with regular or surrogate public data (see Figure <ref type="figure">14</ref>). However, when the ACS dataset is sub-sampled to a smaller dataset (e.g., 5% of the records), we observe a similar pattern regarding the usefulness the traditional and surrogate public data as with the EDAD and WE datasets. See Section 5.1 for a further analysis of the role of dataset size. See Appendix E.1 for an in-depth treatment of all pretraining results.</p><p>Task 2: Hyperparameter Tuning for Synthetic Data Generation. No single surrogate publicdata strategy dominates across all evaluation criteria (Figure <ref type="figure">3</ref>; Tables <ref type="table">1</ref><ref type="table">2</ref><ref type="table">3</ref>). Each generation method achieves superior performance on a subset of metrics -whether predictive accuracy, preservation of pairwise correlations, or marginal distribution fidelity. Pareto-frontier analysis suggests that the critical determinant is the extent to which a surrogate captures higher-order dependence structure; even the Arbitrary baseline appears on the frontier even though it does not encode similar statistical relationship of the private data (for evidence, see Figure <ref type="figure">3</ref>; Table <ref type="table">3</ref>). Nonetheless, the LLM-generation methods (CSV and Agent) exhibit the most favorable overall trade-offs, underscoring their utility for hyperparameter tuning of DP synthetic data generators. See Appendix E.2 for an in-depth treatment of all hyperparameter tuning results.</p><p>Remark: Why might Arbitrary be performant here? A plausible explanation is that PrivBayes benefits from BN-structured surrogates when prioritizing k-way interactions; even randomly param-eterized BNs can provide a coarse proxy for how distributional structure matters during public-space hyper-parameter selection. In contrast, Arbitrary's misspecified marginals undercut classifier pretraining (Task 1) and offer no consistent gains for privacy-utility curve matching <ref type="bibr">(Task 3)</ref>.</p><p>Task 3: Privacy-utility Estimation for Synthetic Data Generation. The story for this task is less clear-cut. While surrogate public data generally provides a reasonable approximation for the privacyutility tradeoff curves, the differences between various generation methods were not pronounced.</p><p>We observed that regular public data provided the best or second-best estimation of the privacyutility tradeoff curve in the vast majority of cases. This observation suggests that data similarity may be an important contributing factor. However, a subsequent analysis (Section 5.2) examining the role of similarity did not reveal a clear pattern that explains this result. See Appendix E.4 for an in-depth treatment of all privacy-utility estimation results.</p><p>5.1 ACS and the Role of Dataset Size .05.1 .2 .5 1 Subsampling -0.05 0.00 0.05 0.10 0.15 0.20 AUC Advantage Public Baseline Arbitrary CSV Agent Without pretraining Figure 4: Mean AUC Advantage of the DP classification model with &#949; = 1 after pretraining for each subsampled dataset, grouped by generation method category. The mean is calculated across the DP finetuning hyperparameter space when best pretraining hyperparameter configuration is chosen for the pretraining step, with 10 runs per hyperparameter configuration.</p><p>For the ACS dataset, we do not observe any benefit from pretraining, either with traditional or surrogate public data (e.g., as Figure <ref type="figure">14b</ref> shows for &#949; = 1). However, a follow-up analysis reveals that this is due to the relatively large size of the dataset. Dataset size is a key factor in differentially private mechanisms, as it directly influences the noise level added to achieve a specific level of privacy protection <ref type="bibr">[38]</ref>. The relatively large size of the ACS dataset partly explains why the benefit of regular public data pretraining appears marginal in, e.g., Table <ref type="table">8a</ref>; as privacy sensitivity scales inversely with dataset size, when the private dataset is sufficiently large, the magnitude of noise necessary for a DP guarantee decreases.</p><p>To investigate this effect, we repeated the full pretraining experiment on four ACS subsets obtained by subsampling at 5%, 10%, 20%, and 50%.</p><p>In these experiments we focus on the AUC advantage at &#949; = 1, where the benefit of public data is most pronounced. Figure <ref type="figure">14b</ref> (in Appendix E.1) shows that with 5% subsampling, the ACS dataset exhibits a similar pattern of performance to the one we found with the EDAD and WE datasets.</p><p>In fact, the LLM-based methods (using Claude Sonnet 3.5) outperformed the traditional public dataset. Figure <ref type="figure">4</ref> presents the relationship between the (subsampled) dataset size and the AUC advantage per generation method category to &#949; = 1. As we examine smaller datasets, the differences we observe align with results on the EDAD and WE datasets. Both the CSV and Agent surrogate datasets perform on average similarly to traditional public data. This observation may also help explain the negative findings reported by Swanberg et al. <ref type="bibr">[100]</ref> regarding the use of LLM-generated public data for DP synthetic data generation on the Adult dataset, which is substantially larger than some of the datasets considered here. We hypothesize that with smaller datasets, LLM-generated public data surrogates could provide some benefit in pretraining differentially private data synthesizers, but leave a closer examination of that DP auxiliary task to future work.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.2">Dataset Similarity May Be Less Important Than You'd Think</head><p>By using public data in DP auxiliary tasks, we implicitly assume statistical similarity to the private, sensitive data. Our results generally back this up; in pretraining (Task 1) and privacy-utility trade-off estimation (Task 3), we observe consistently better traditional public data performance. To explore whether the traditional public data dominance (and the relative performance rankings of the surrogate public data) could be explained by dataset similarity, we measure the similarity between all datasets using two common metrics from the DP literature (see Appendix G.1): Total Variation Distance (TVD) and average error across all 3-way marginal queries (3WM) <ref type="bibr">[73,</ref><ref type="bibr">80]</ref>.</p><p>Comparing Private vs. Public. The first dataset similarity question we ask is: how similar is a public data variant to the true, private data? Both traditional and surrogate private vs. public data similarity results are shown in Table <ref type="table">25</ref>. In general, our metrics suggest that the traditional public dataset and the Univariate baseline (recall, this baseline samples independently with a little noise from the true private distribution) are most similar to the private data. For EDAD and WE datasets, we can explain the lower overall similarity scores due to their higher dimensionality (defined as the Cartesian product of possible unique variable values; see the "&#215; Dims" column in Table <ref type="table">1</ref>); the dataset distance is exacerbated by sparsity (particularly for TVD). However, even accounting for the limitations of these metrics, we did not observe a clear relationship between the similarity rankings of public datasets and their usefulness rankings in the pretraining and privacyutility tasks. Our explanatory hypothesis: common similarity metrics, like TVD and 3WM, may not adequately capture dataset characteristics relevant to the DP auxiliary tasks we frame. We leave further exploration of suitable metrics to future research.</p><p>Comparing Among (Traditional or Surrogate) Public Data. The second dataset similarity question we ask is: how similar are public data variants to each other, and does this partially explain their relative performance rankings? To this end, we compared similarity metrics among traditional and surrogate public data, with heatmap plots provided in Appendix G. The most consistent pattern observed across datasets and metrics is the strong similarity between Agent pairs using the same LLM but differing only in mixing methods (Unif. vs. Max Cov.) (this is barring TVD for EDAD and WE, where most entries are zero due to the aforementioned dimensionality constraints). This pattern extends to similarities between the overall mixing datasets and individual Agent datasets (with the exception of the Llama datasets on EDAD). This is expected since these pairs share the same underlying source of sampled records. Interestingly, we did not find stable similarities across generated data between different LLMs within either Agent or CSV methods, or between the same LLM across these two methods. Again, this could be an artifact of the metrics we use, but we leave a deeper exploration of this for future work.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6">Related Work</head><p>Public data in differential Privacy. Historically, the notion of using surrogate (independently generated) public data for downstream private learning appears in earlier work on e.g. on importanceweighted DP learning <ref type="bibr">[60]</ref>, which we view as philosophically aligned with our setup. Extensive prior work has then further shown how public data can improve DP machine learning via pretraining followed by private fine-tuning, particularly in NLP and vision tasks <ref type="bibr">[106,</ref><ref type="bibr">7,</ref><ref type="bibr">118,</ref><ref type="bibr">44,</ref><ref type="bibr">43,</ref><ref type="bibr">53,</ref><ref type="bibr">20,</ref><ref type="bibr">104,</ref><ref type="bibr">63]</ref>. Public data is integrated directly into differentially private computations for tasks such as private estimation <ref type="bibr">[15]</ref>, statistical queries <ref type="bibr">[12,</ref><ref type="bibr">71]</ref>, and predictive learning <ref type="bibr">[11,</ref><ref type="bibr">111,</ref><ref type="bibr">13,</ref><ref type="bibr">61,</ref><ref type="bibr">123,</ref><ref type="bibr">85,</ref><ref type="bibr">14,</ref><ref type="bibr">48,</ref><ref type="bibr">86,</ref><ref type="bibr">110,</ref><ref type="bibr">17,</ref><ref type="bibr">76]</ref>. Differentially private fine-tuning of pretrained LLMs with DP-SGD <ref type="bibr">[1]</ref> has been employed to generate synthetic data for downstream tasks <ref type="bibr">[67,</ref><ref type="bibr">119,</ref><ref type="bibr">8,</ref><ref type="bibr">115]</ref> and is comprehensively reviewed in Cummings et al. <ref type="bibr">[31]</ref>.</p><p>Related LLM approaches. Contemporary work by Swanberg et al. <ref type="bibr">[100]</ref> is closely related to ours; they evaluate LLM-generated public data in a single setting for public pretraining of private synthetic data mechanisms. However, their work focuses on a narrow experimental setup on pretraining private synthetic data models; in contrast, our study assesses surrogate data for several DP auxiliary tasks, spans multiple datasets and metrics, and includes memorization tests <ref type="bibr">[18]</ref>.</p><p>Generating tabular data. Transformer-based neural models have been applied to generate synthetic tabular data either by training models from scratch <ref type="bibr">[97,</ref><ref type="bibr">49,</ref><ref type="bibr">122]</ref> or by pretraining a tabular foundation model <ref type="bibr">[77]</ref>. Pretrained LLMs have also been adapted via fine-tuning <ref type="bibr">[19]</ref> or in-context learning <ref type="bibr">[96,</ref><ref type="bibr">66]</ref>, but these methods are unsuitable since they condition directly on sensitive data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7">Limitations</head><p>The overarching goal of this work is to address a significant barrier to the real-world adoption of differential privacy <ref type="bibr">[38]</ref>. Enabling more widespread deployment of differential privacy, when appropriate, can promote the protection of individuals privacy <ref type="bibr">[31]</ref>. However, our work does have several risks and limitations. First, there is a risk that LLM memorization may lead to overly optimistic performance estimates (we attempted to mitigate this risk by carefully checking for evidence of memorization, see Appendix D.2.4).</p><p>Second, the normative implications of employing LLMs to generate surrogate public data should be carefully analyzed in this context. Recent work by Tram&#232;r et al. <ref type="bibr">[107]</ref> cautions against treating web-scraped LLM training data as "public" or non-sensitive. Traditionally, differentially private algorithms have assumed data is either fully private (restricted) or fully public (freely available and safe to reuse). However, Tram&#232;r et al. <ref type="bibr">[107]</ref> emphasize a messier reality; social media and other sources of personally identifiable information, for example, may be both accessible to language models for training data and contain sensitive information specific to individuals. When an LLM is trained on such data, it may memorize fragments of it; regurgitating these private fragments could be interpreted as a privacy violation. Indeed, even if a final model is fine-tuned under DP constraints, privacy violations may originate from the pretrained model (e.g., a base model memorized private details during pretraining, and a subsequent DP fine-tuning step does not noise those probabilities sufficiently to obfuscate). This undermines trust, as an individual may be told that the entire pipeline is "privacy preserving," yet see their personal data re-emerge in the final models outputs.</p><p>We carefully position our work under the paradigm shift identified by Tram&#232;r et al. <ref type="bibr">[107]</ref>. Using LLMs to emulate expert-driven data-generating processes risks inadvertently exposing sensitive information that is publicly available, as mediated by the LLM. Thus, we propose that best practice is to report empirical measurements of memorization levels. We do this by leveraging work by Bordt et al. <ref type="bibr">[18]</ref> on verbatim memorization of tabular data by LLMs; see Appendix D.2.4. Additionally, we report on datasets (see Table <ref type="table">1</ref>) that post-date LLM training (for the models we evaluate; see Table <ref type="table">2</ref>). Choosing tasks where the LLMs prior knowledge is outdated or non-existent demonstrates performance on truly unseen data <ref type="bibr">[29]</ref>. We stress the importance of communicating these nuances, and of reporting, to the best of one's knowledge/ability, the empirical level of memorization and the potential LLM data regurgitation risks when presenting these methods.</p><p>Additionally, given substantial evidence that LLMs encode biases <ref type="bibr">[41]</ref>, these biases could be reflected in the generated data -either implicitly in CSV generation or explicitly via the causal relationships in the Agent-based approach. For instance, a stereotypical correlation could persist through pretraining and DP fine-tuning, ultimately resulting in an unfair classifier. We leave a detailed investigation of these issues for future work.</p><p>Finally, we note that our methods require minimal metadata per column: (1) name, (2) one-line description (if available), and (3) categorical domain or plausible/known numeric range. Such schemas are extremely common in social-science surveys (ICPSR, UKDS) and medical data models (FHIR, OMOP). We defer an investigation of, e.g., noisy logs or sensor data that lack a coherent schema to future work. The CSV and Agent natively support continuous values (validator-checked CSV; parametric numeric nodes in Agent). We bin continuous columns in our experiments to align with current synthesizers; principled discretization design, potentially with a small DP budget, remains a possible extension. Because surrogates are mechanism-agnostic, they could easily be extended to support federated or local DP pipelines; we leave empirical validation to future work.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="8">Conclusion</head><p>In this work, we asked whether LLMs can be used to generate effective surrogate public data for solving DP auxiliary tasks in settings where traditional public tabular data is limited or unavailable. Each approach we considered leveraged schema-level metadata to generate surrogate public data in the same domain as the private, sensitive data. We considered LLM data generation methods like directly prompting for tabular CSV records, and through an Agent that constructs an SCM over the schema variables using the LLM as an expert prior. Our evaluations demonstrated that LLM-generated public data surrogates can be used to significantly improve the DP auxiliary task of private classifier pretraining with public data. The LLM-generated public data surrogates were also useful for the tasks of hyperparameter tuning and privacy/utility tradeoff estimation, albeit with less impressive performance relative to a strong baseline (Arbitrary). Overall, our results provide an affirmative answer: for the DP auxiliary tasks we considered, generating surrogate public data with LLMs can overcome tabular public data scarcity.</p><p>NeurIPS Paper Checklist</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="1.">Claims</head><p>Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?</p><p>Answer: <ref type="bibr">[Yes]</ref> Justification: Our intro / methods sections discuss contributions and approaches, and our results section (and extensive appendices) support our claims.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the abstract and introduction do not include the claims made in the paper.</p><p>&#8226; The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.</p><p>&#8226; The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.</p><p>&#8226; It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="2.">Limitations</head><p>Question: Does the paper discuss the limitations of the work performed by the authors?</p><p>Answer: <ref type="bibr">[Yes]</ref> Justification: We provide an extensive limitations section that highlights the main risks and a prior position paper that is relevant to our work Guidelines:</p><p>&#8226; The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.</p><p>&#8226; The authors are encouraged to create a separate "Limitations" section in their paper.</p><p>&#8226; The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.</p><p>&#8226; The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.</p><p>&#8226; The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.</p><p>&#8226; The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.</p><p>&#8226; If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.</p><p>&#8226; While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="3.">Theory assumptions and proofs</head><p>Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?</p><p>Answer: <ref type="bibr">[NA]</ref> Justification: The paper does not provide formal proofs.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not include theoretical results.</p><p>&#8226; All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.</p><p>&#8226; All assumptions should be clearly stated or referenced in the statement of any theorems.</p><p>&#8226; The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.</p><p>&#8226; Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.</p><p>&#8226; Theorems and Lemmas that the proof relies upon should be properly referenced.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="4.">Experimental result reproducibility</head><p>Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?</p><p>Answer: <ref type="bibr">[Yes]</ref> Justification: We link to the open source package (via github) that can be used to reproduce all of the experimental results in this work.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not include experiments.</p><p>&#8226; If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.</p><p>&#8226; If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.</p><p>&#8226; Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.</p><p>&#8226; While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.</p><p>In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="5.">Open access to data and code</head><p>Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?</p><p>Answer: [Yes] Justification: See above justification.</p><p>Guidelines:</p><p>&#8226; The answer NA means that paper does not include experiments requiring code.</p><p>&#8226; Please see the NeurIPS code and data submission guidelines (<ref type="url">https://nips.cc/</ref> public/guides/CodeSubmissionPolicy) for more details. &#8226; While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).</p><p>&#8226; The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (<ref type="url">https:  //nips.cc/public/guides/CodeSubmissionPolicy</ref>) for more details. &#8226; The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.</p><p>&#8226; The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.</p><p>&#8226; At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).</p><p>&#8226; Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="6.">Experimental setting/details</head><p>Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?</p><p>Answer: [Yes]</p><p>Justification: We enumerate all details / hyperparameters in the appendices.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not include experiments.</p><p>&#8226; The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.</p><p>&#8226; The full details can be provided either with the code, in appendix, or as supplemental material.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="7.">Experiment statistical significance</head><p>Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?</p><p>Answer: [Yes] Justification: Our extensive experiments have error bars and report over many runs.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not include experiments.</p><p>&#8226;</p><p>The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.</p><p>&#8226; The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).</p><p>&#8226; The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)</p><p>&#8226; The assumptions made should be given (e.g., Normally distributed errors).</p><p>&#8226; It should be clear whether the error bar is the standard deviation or the standard error of the mean.</p><p>&#8226; It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.</p><p>&#8226; For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).</p><p>&#8226; If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="8.">Experiments compute resources</head><p>Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?</p><p>Answer: [Yes]</p><p>Justification: Details on the datasets and experiments (compute, etc.) are in the paper appendix.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not include experiments.</p><p>&#8226; The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.</p><p>&#8226; The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.</p><p>&#8226; The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="9.">Code of ethics</head><p>Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics <ref type="url">https://neurips.cc/public/EthicsGuidelines</ref>?</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Answer: [Yes]</head><p>Justification: After reviewing, we believe we conform.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.</p><p>&#8226; If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.</p><p>&#8226; The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="10.">Broader impacts</head><p>Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?</p><p>Answer: [Yes]</p><p>Justification: Yes, in our limitations section we address.</p><p>Guidelines:</p><p>&#8226; The answer NA means that there is no societal impact of the work performed.</p><p>&#8226; If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.</p><p>&#8226; Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.</p><p>&#8226; The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.</p><p>&#8226; The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.</p><p>&#8226; If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="11.">Safeguards</head><p>Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?</p><p>Answer: <ref type="bibr">[Yes]</ref> Justification: All data used in this project was already public domain.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper poses no such risks.</p><p>&#8226; Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.</p><p>&#8226; Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.</p><p>&#8226; We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.</p><p>12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?</p><p>Answer: [Yes]</p><p>Justification: They are cited and linked to appropriately.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not use existing assets.</p><p>&#8226; The authors should cite the original paper that produced the code package or dataset.</p><p>&#8226; The authors should state which version of the asset is used and, if possible, include a URL.</p><p>&#8226; The name of the license (e.g., CC-BY 4.0) should be included for each asset.</p><p>&#8226; For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.</p><p>&#8226; If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.</p><p>&#8226; For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.</p><p>&#8226; If this information is not available online, the authors are encouraged to reach out to the asset's creators.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="13.">New assets</head><p>Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?</p><p>Answer: <ref type="bibr">[Yes]</ref> Justification: All data used in this project was already public domain and we have documented our processing of it in the repo.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not release new assets.</p><p>&#8226; Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.</p><p>&#8226; The paper should discuss whether and how consent was obtained from people whose asset is used.</p><p>&#8226; At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="14.">Crowdsourcing and research with human subjects</head><p>Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?</p><p>Answer: <ref type="bibr">[NA]</ref> Justification: No human subjects.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.</p><p>&#8226; Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.</p><p>&#8226; According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: No human subjects.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.</p><p>&#8226; Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.</p><p>&#8226; We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.</p><p>&#8226; For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head n="16.">Declaration of LLM usage</head><p>Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>Answer: [Yes]</head><p>Justification: The paper describes the usage of LLMs to generate public surrogate data for DP auxiliary tasks. We extensively discuss this use case.</p><p>Guidelines:</p><p>&#8226; The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.</p><p>&#8226; Please refer to our LLM policy (<ref type="url">https://neurips.cc/Conferences/2025/LLM</ref>) for what should or should not be described.</p><p>Contents 1 Introduction 2 Preliminaries 3 Producing Public Data Surrogates 4 Evaluation Framework 5 Results 6 Related Work 7 Limitations 8 Conclusion Acknowledgement A Future Work B Additional Related Work C Details of Surrogate Public Data Generation D Details of Evaluation Framework E Additional Result Discussion F Additional Detailed Results G Details of Dataset Similarity H Compute and Resources</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>A Future Work</head><p>The strong performance of the Arbitrary method in hyperparameter tuning is intriguing, as it suggests that finding good-enough hyperparameter configuration might depend on the record domain and the synthetic data generators, and not necessarily on the private data. This raises questions about potential theoretical justifications for this observation.</p><p>The fact that traditional public data often performs best for privacy/utility trade-off estimation would lead us to believe that dataset similarity plays an important role for this task. We hypothesize that the two similarity metrics used in this work, while being natural candidates, may not adequately capture dataset characteristics relevant to estimating the behavior of data synthesizers across privacy budget settings. Identifying metrics that better predict which surrogate data provides accurate tradeoff estimations would be beneficial. Such a metric could enable, e.g., the exponential mechanism <ref type="bibr">[82]</ref> to select similar datasets (or combinations of datasets), if such a metric had low sensitivity with respect to the private dataset.</p><p>Several additional DP auxiliary tasks remain unexplored in our study, such as using public data for seeding synthetic data generation <ref type="bibr">[100]</ref> and assessing the success rate of privacy attacks as a function of &#949; <ref type="bibr">[32]</ref>. We leave these avenues for future research.</p><p>We propose three approaches to improve the quality of surrogate data produced by Agent-based methods, making it more closely resemble private data. First, subject matter experts can review and refine the generated SCM to better encode experts' domain knowledge. Second, Retrieval-Augmented Generation (RAG) <ref type="bibr">[70]</ref> could be beneficial to surface specific knowledge from scientific literature, enabling the model to incorporate both accurate causal relationships and their quantitative parameters as established in peer-reviewed research. Third, recent advancements in reasoning LLMs <ref type="bibr">[99,</ref><ref type="bibr">58,</ref><ref type="bibr">34]</ref> may enhance LLMs' ability to consider causal relationships.</p><p>Finally, some recent work on Sequence Driven Structural Causal Models (SD-SCMs) shows how to simulate counterfactual outcomes and treatment scenarios that are often inaccessible in sensitive datasets (by allowing an LLM to specify structural equations implicitly, given a topological order and a specific prompting structure) <ref type="bibr">[22]</ref>. Similarly to the surrogate public data approaches explored in this paper, the SD-SCM approach does not require access to a downstream private dataset of interest; instead, it only requires a schema over the data to be generated, and a user to specify the prompting structure and topological order over variables (which could be generated e.g., by the first few steps of the Agent procedure given in Figure <ref type="figure">7</ref>). There may be many potential uses for SD-SCM generated surrogate public data for private causal algorithms; for example, we believe that future work could explore how it can be used to improve the performance of hyperparameter tuning for private causal effect estimators.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>B Additional Related Work</head><p>Public data in differential privacy Empirical evidence demonstrates that public data can improve the performance of differentially private machine learning models through a two-stage approach: pretraining on public data followed by differentially private fine-tuning on sensitive data. This approach has been extensively studied across NLP and vision tasks <ref type="bibr">[106,</ref><ref type="bibr">7,</ref><ref type="bibr">118,</ref><ref type="bibr">44,</ref><ref type="bibr">43,</ref><ref type="bibr">53,</ref><ref type="bibr">20]</ref>.</p><p>Ganesh et al. <ref type="bibr">[42]</ref> identify two phases in neural network optimization within non-convex loss landscapes. The first locates an optimal basin, where public data suffices and using the privacy budget is unnecessary. The second performs local optimization within that basin; here, if the public and target distributions differ -as they often do -consuming privacy budget to update weights with sensitive data is beneficial. Supporting this, Thaker et al. <ref type="bibr">[104]</ref> show that public pretraining outperforms fully private training in vision tasks, even under significant distribution shifts. This advantage holds even when private fine-tuning is limited to the final layer, as in Ke et al. <ref type="bibr">[63]</ref>.</p><p>Another research direction incorporates public data directly into differentially private computations, rather than treating it as a separate preprocessing step. This approach spans private estimation <ref type="bibr">[15]</ref>, statistical queries <ref type="bibr">[12,</ref><ref type="bibr">71]</ref>, and learning and optimization <ref type="bibr">[11,</ref><ref type="bibr">111,</ref><ref type="bibr">13,</ref><ref type="bibr">61,</ref><ref type="bibr">123,</ref><ref type="bibr">85,</ref><ref type="bibr">14,</ref><ref type="bibr">48,</ref><ref type="bibr">86,</ref><ref type="bibr">110,</ref><ref type="bibr">17,</ref><ref type="bibr">76</ref>]. An emerging line of research finetunes pretrained, open-source LLMs on private, sensitive data with DP-SGD <ref type="bibr">[1]</ref> to generate training data for downstream models, such as classifiers or other LLMs <ref type="bibr">[67,</ref><ref type="bibr">119,</ref><ref type="bibr">8,</ref><ref type="bibr">115]</ref>. For a broader survey on recent advances in privacy research, see <ref type="bibr">[31]</ref>.</p><p>Finally, contemporary work by Swanberg et al. <ref type="bibr">[100]</ref> is closely related to ours, but with three key differences. First, while they evaluate LLM-generated public data in a single experimental setting (for public pretraining of private synthetic data mechanisms), we assess its utility across several DP auxiliary tasks -including hyperparameter tuning for synthetic data generation, privacy/utility tradeoff estimation, and private classifier pretraining. Second, our evaluation is broader in scope, incorporating multiple datasets (with different data-origins), diverse metrics and additional baselines / methods for leveraging an LLM to produce surrogate public data. Third, we designed our experiments to mitigate the risk of positive results due to memorization, including an explicit test based on Bordt et al. <ref type="bibr">[18]</ref>, and provide results and analysis to assess the impact of data leakage on the performance of our methods.</p><p>Generating tabular data with LLMs Transformer-based models can be used to generate synthetic samples from tabular data. The fundamental approach involves treating each record as a "sentence" for the transformer architecture to process. Two overall strategies exist: training transformers from scratch specifically for tabular data and adapting pretrained LLMs for tabular generation tasks. For the first strategy, one variant trains a transformer on an individual dataset or distribution to produce synthetic records <ref type="bibr">[97,</ref><ref type="bibr">122,</ref><ref type="bibr">49,</ref><ref type="bibr">122]</ref>; another variant pretrains a general tabular foundation model on multiple datasets and then adapts this model to novel unseen datasets through in-context learning <ref type="bibr">[77]</ref>. The second strategy uses existing pretrained LLMs, adapting them for tabular data generation through either fine-tuning <ref type="bibr">[19]</ref> or in-context learning <ref type="bibr">[96,</ref><ref type="bibr">66]</ref>. These methods cannot be directly applied to our setting, as we only consider DP auxiliary tasks (e.g., pretraining, hyperparameter tuning) that do not consume privacy budget. Both approaches condition on sensitive data and thus require accounting for privacy loss.</p><p>Recent work has explored the potential of LLMs for causal modeling tasks, including pairwise causal discovery, causal model generation, and counterfactual reasoning <ref type="bibr">[65,</ref><ref type="bibr">27]</ref>. While causality itself is not the primary focus of our project, the ability to produce plausible causal models is highly relevant since causal models are also generative, capable of producing realistic records. LLM-based causal model discovery methods can operate either with metadata alone (using only dataset descriptions and schema) <ref type="bibr">[108,</ref><ref type="bibr">75,</ref><ref type="bibr">74,</ref><ref type="bibr">121,</ref><ref type="bibr">33,</ref><ref type="bibr">22]</ref> or with additional observations <ref type="bibr">[2,</ref><ref type="bibr">68]</ref> -with the metadataonly approach being particularly relevant to our project as we have no access to observations. Most literature in this area that operates without observations focuses solely on discovering causal graphs -descriptions of causal dependencies without specifying conditional distributions. However, <ref type="bibr">[22]</ref> extends this approach by adding a second step that prompts LLMs with the topological order over variables, embedded in a prompt structure, to generate records directly.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C Details of Surrogate Public Data Generation</head><p>C.1 Baselines Before discussing the LLM-based approach, we present a series of baseline generation processes to systematically evaluate which aspects of public data characteristics are useful for differential privacy tasks: pretraining, hyperparameter tuning, and estimating the privacy-utility trade-off. The baselines differ in statistical structure and in the information available about the private data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.1.1 Uniform Distribution over the Domain</head><p>The dimensionality of the data plays a critical role in differentially private algorithms <ref type="bibr">[79,</ref><ref type="bibr">93]</ref>, as it could affect, for example, the magnitude of noise introduced to satisfy DP or the ratio between signal and that noise (e.g., when tuning data synthesizers like PrivBayes or AIM). This Uniform distribution baseline captures the scenario where we have no prior knowledge about the underlying data distribution beyond the schema itself by using the maximum entropy probability distribution <ref type="bibr">[59]</ref>: for each record, Uniform samples i.i.d. from either the set of possible values (for categorical columns) or the specified range (for continuous columns), both given in the schema.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.1.2 Univariate Distribution</head><p>Beyond knowledge of the record domains, organizations and researchers might have access to prior information about the univariate distributions of individual columns, either precisely or approximately. This prior knowledge is available in cases where organizations may have released various statistical measures of private data, such as histograms, means, medians, and standard deviations, with or without differential privacy <ref type="bibr">[94,</ref><ref type="bibr">52]</ref>. As a facsimile for data generated with knowledge of the distributions along individual columns, the Univariate baseline samples independently from each column according to the empirical univariate distribution drawn directly from the private data.</p><p>To make this baseline more realistic -assuming only an approximate PDF (e.g., the distributions "shape") is known -we round the probabilities to two decimal places, normalize to 1, and rescale during sampling.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.1.3 Arbitrary Distribution</head><p>The previous two baselines are limited by column independence in their sampling, preventing them from capturing complex statistical structures needed for higher-order analysis and predictive tasks <ref type="bibr">[93]</ref>. To isolate the role of structural dependencies in our DP auxiliary tasks with surrogate public data, we consider whether only capturing the existence of relationships between columns could make surrogate public data a useful prior. To test this, we generate an arbitrary dataset from a random but structured distribution that adheres to the schema.</p><p>Algorithm 1 details the full Arbitrary baseline procedure; here, we provide a high-level overview of the two-step generation process. First, we construct a random Directed Acyclic Graph (DAG) representing a Bayesian network over the column variables. The DAG is built sequentially, with each new node potentially connecting to any previously added nodes, subject to a maximum in-degree (here we used 5). This ensures a structured yet arbitrary dependency pattern between variables. Second, we parameterize the network by sampling conditional probability tables for each node. For a given node, we use a Dirichlet distribution with concentration parameter &#945; = 1 to generate probability distributions for each configuration of its parent variables. Specifically, for each parent value combination, we sample a categorical distribution from the k-simplex, where k is the cardinality of the node's domain. This yields a distribution with meaningful dependencies (e.g., correlations) while remaining entirely independent of the true empirical distribution of the private data.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.2 CSV Direct Generation</head><p>We evaluate a direct approach to data generation using LLMs. The generation process involves prompting the LLM to create CSVs -tabular records that adhere to the schema while following specific guidelines <ref type="bibr">[6]</ref>. These guidelines instruct the model to ensure realistic value distributions and relationships between fields, maintain real-world patterns and constraints, and incorporate edge cases at frequencies that mirror their natural occurrence. Similarly to the other surrogate public data methods we evaluate, this approach operates without access to the private dataset, relying solely on the LLM's pretrained knowledge.</p><p>To ensure data quality, each generated record is validated against the schema, and only valid records are retained. Due to context window limitations and API constraints, the generation process is executed in multiple batches until the desired number of records is obtained <ref type="bibr">[87,</ref><ref type="bibr">9,</ref><ref type="bibr">105]</ref>. Note that, due to the autoregressive nature of LLMs (see e.g., Section 2), records within the same generation batch are not sampled independently, in contrast to the baseline methods.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>C.3 Agent (State Machine) Approach</head><p>As a final approach, we employ a multi-step, Agent-based process to elicit a structural causal model (SCM) from an LLM given only text-based access through prompts and responses. Our goal is to arrive at a coherent directed acyclic graph (DAG) that captures the inter-dependencies among variables in the schema, along with associated structural equations (e.g., the actual distributional parameters, probabilities, etc.). Each step concludes with an automated validation of the LLM's output; so, if any contradictions or omissions are detected, the Agent (implemented as a state machine, see Figure <ref type="figure">7</ref>) automatically refines our prompt and re-queries the LLM.</p><p>First, we prompt the LLM to 1list out all variables (keys) from the provided schema, ensuring the response exactly matches the schemas variable set. Next, we ask it to 2propose realistic consistency constraints among these variables; these constraints should capture domain knowledge such as permissible value ranges (e.g., "age must be at least 0") or logical relationships (e.g., "an individual who is 10 years old must have fewer than 10 years of education"). We then instruct the LLM to 3 identify a subset of variables that can serve as the "root nodes" in a causal graph, typically those deemed exogenous or less likely to be influenced by other variables in the schema. From there, the LLM proposes parentchild relationships 4from root nodes to non-root nodes, and then 5among all remaining variables, 6ensuring no cycles are introduced so that the final structure is a DAG (which we validate with a graph library to confirm it contains all variables exactly once and remains acyclic <ref type="bibr">[50]</ref>).</p><p>Having obtained a DAG, we prompt the LLM to 7map each variable to a structural equation that references its parents. For instance, if a node depends on two parents, the LLM might generate a formula specifying a probabilistic distribution conditional on parent values. These structural equations encode marginal distributions for root variables and conditional distributions for their descendants. Sometimes the structural equations are not fully specified (e.g., the probability parameter in Bernoulli distribution is parameterized), so we instruct the LLM to 8assign values to all parameters. Then, 9we combine the DAG and structural equations automatically into a single code snippet (we use the Pyro library <ref type="bibr">[16]</ref>), which lets us generate synthetic data automatically. Finally, we ask the LLM to amend the Python code to 10 enforce the range or valid values for each column, and 11 include the constraints elicited at the beginning of the interaction. This entire interaction is a stateful, automatic, closed loop, allowing the LLM to act on its own as an "expert" to design a plausible causal model solely from schema level information (containing short descriptions of each variable), without a need to inspect any real-world sensitive records.</p><p>To extend this approach from a "single expert" to a "panel of experts," we execute the complete generation workflow multiple times to produce a collection of datasets, inspired by prior work on "self-consistency" prompting methods <ref type="bibr">[112]</ref>. These datasets are then combined to yield a single mixed dataset using two approaches. The first approach, Unif., involves uniform sampling of records across all generated datasets. The second approach, Max Cov., solves the Facility Location submodular problem <ref type="bibr">[111]</ref> by finding a subset of datasets that maximizes the sum of pairwise Total Variation similarities. This optimization selects a subset of datasets that aims to represent the space of all generated datasets <ref type="bibr">[111]</ref>. Then, similarly to the Unif. approach, we sample records uniformly from the selected datasets.</p><p>One important advantage of agent-generated SCMs is that domain experts can modify the causal structure, structural equations, and constraints based on their expertise, scientific literature, and common sense. We leave this for future work. { . . . "RELACT " : { " d e s c r i p t i o n " : " Main l a b o u r m a r k e t a c t i v i t y s t a t u s " , " d t y p e " : " i n t 6 4 " , " v a l u e s " : { " 1 " : " Employed " , " 2 " : " Unemployed " , " 3 " : " R e t i r e d " , " 4 " : " S t u d e n t " , " 5 " : " U n a b l e t o work " , " 6 " : " Doing u n p a i d s o c i a l work o r c h a r i t a b l e a c t i v i t i e s &#8617;&#8594; " , " 7 " : " O t h e r i n a c t i v e p e r s o n " } } , "CERTIG " : { " d e s c r i p t i o n " : " D e g r e e o f d i s a b i l i t y " , " d t y p e " : " i n t 6 4 " , " v a l u e s " : { " 1 " : "0 -32%" , " 2 " : "33 -44%" , " 3 " : "45 -64%" , " 4 " : "65 -74%" , " 5 " : "75% o r more " , " 6 " : " Not known " } } , " AUDI_7_1 " : { " d e s c r i p t i o n " : " Has s i g n i f i c a n t d i f f i c u l t y h e a r i n g a &#8617;&#8594; c o n v e r s a t i o n w i t h s e v e r a l p e o p l e w i t h o u t a h e a r i n g &#8617;&#8594; a i d " , " d t y p e " : " i n t 6 4 " , " v a l u e s " : { " 1 " : " Yes " , " 2 " : "No" } } , . . . } Figure <ref type="figure">5</ref>: Excerpt from the schema of the EDAD dataset (Spanish disability, autonomy, and dependency survey) <ref type="bibr">[56]</ref>.</p><p>Algorithm 1 Random Bayesian network generation for the arbitrary dataset. 1Schema start 2Elicit constraints 3Root nodes 6DAG 5Non-root to non-root edges 4Root nodes to non-root edges 7Structural equations 8Parameters 9Pyro code 12 Sample 11 Enforce constraints 10 Enforce range failed failed failed failed failed failed failed failed failed failed failed success success success success success success success success success success success Figure 7: State machine for the SCM Agent showing state transitions. Each state can transition to itself upon failure or advance to the next state upon success, following a zigzag pattern.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D Details of Evaluation Framework</head><p>Category Metric Description Marginals Total Variation Distance Distance between the joint distributions of the original and synthetic datasets. Max 3-Way Marginal Error Maximum absolute difference error for 3-way marginals between original and synthetic datasets, normalized by dataset size. Avg. 3-Way Marginal Error Average absolute difference error for 3-way marginals between original and synthetic datasets, normalized by dataset size and query count. Max Binarized Marginal Error Maximum absolute difference error for 3-way marginals after thresholding continuous variables to binary values, normalized by dataset size. Avg. Binarized Marginal Error Average absolute difference error for 3-way marginals after thresholding continuous variables to binary values, normalized by dataset size and query count. Correlations Max Pearson Correlation Diff Maximum absolute difference between Pearson correlation coefficients of original and synthetic datasets. Avg. Pearson Correlation Diff Average absolute difference between Pearson correlation coefficients of original and synthetic datasets. Max Cramer's V Diff Maximum absolute difference between Cramer's V correlation coefficients of original and synthetic datasets.</p><p>Avg. Cramer's V Diff Average absolute difference between Cramer's V correlation coefficients of original and synthetic datasets.</p><p>Classification Error Rate Diff Difference in classification error rates between models trained on original vs. synthetic data and evaluated on the same test set.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>AUC Diff</head><p>Difference in Area Under the ROC Curve (AUC) between models trained on original vs. synthetic data and evaluated on the same test set.</p><p>Table <ref type="table">3</ref>: Overview of quality evaluation metrics for a synthetic dataset against the original dataset. All metrics range from 0 to 1, with lower values indicating better synthetic data quality.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1 Tasks D.1.1 Task 1: Model Pretraining for Classification</head><p>A common practice in machine learning with DP is to first pretrain a model on public data (incurring no privacy loss) before fine-tuning it privately on sensitive data (using e.g., DP-SGD, incurring fixed (&#949;, &#948;)-privacy loss). We apply this method to evaluate surrogate public data for binary classification tasks on tabular data (recall that this is a less common setting than public pretraining with image data <ref type="bibr">[42,</ref><ref type="bibr">104]</ref>, due to a general lack of publicly available priors for tabular datasets).</p><p>We divided public and private datasets into train, validation, and test subsets using a 72 : 8 : 20 ratio, and used an FTTransformer deep neural attention based classification model architecture (Gorishniy et al. <ref type="bibr">[46]</ref>; see Appendix D.3.1 for more details). Our classification evaluation framework follows three steps: (1) standard pretraining, updating model weights with gradients calculated from (surro-gate) public data;</p><p>(2) DP fine-tuning on private training data; and (3) performance assessment on the private test data. For comparison, we include a control condition that omits the pretraining phase. We measure classification performance using AUC metric and ensure balanced datasets by downsampling the majority class to match the minority class size. We also consider an AUC Advantage metric, which we define as the difference in AUC between models with public data pretraining and a model without pretraining, which directly quantifies the incremental benefit provided by pretraining before private finetuning.</p><p>To account for the multiple hyperparameters in both pretraining and fine-tuning stages, we conduct a comprehensive grid search, further running each configuration 10 times to mitigate variations inherent to differential privacy training and model initialization. We analyze results using two complementary approaches: averaging performance across all hyperparameter combinations, and simulating a real-world scenario by selecting the optimal pretraining hyperparameters based on public validation data before averaging results across fine-tuning hyperparameters. Refer to Appendix D.4 for the complete hyperparameter space details.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.2 Task 2: Hyperparameter Tuning for Synthetic Data</head><p>Hyperparameters play an important role in training machine learning models, especially when differential privacy is involved <ref type="bibr">[89]</ref>. While selecting the best performing hyperparameters in the nonprivate setting can be done with many model training runs using a validation split or cross-validation, this is not feasible in a straightforward manner with differential privacy due to the privacy loss incurred on each run. Public data may be helpful in this case, allowing researchers to run multiple experiments without consuming the privacy loss budget <ref type="bibr">[57,</ref><ref type="bibr">26]</ref>.</p><p>To assess the usefulness of surrogate public data for this DP auxiliary task, we run a large-scale DP synthetic data evaluation across multiple dimensions: (1) datasets (including private and public splits, and various public data surrogates); (2) privacy loss budget &#949;; (3) different DP synthetic data generators (see Section D.3.2; GEM <ref type="bibr">[72]</ref>, AIM <ref type="bibr">[80]</ref>, PrivBayes <ref type="bibr">[120]</ref>); and (4) their associated hyperparameter spaces. For each configuration, we fit a synthetic data generator and produce a synthetic dataset of the same size as the original, private data. We then evaluate across a variety of metrics, which fall into three general categories: marginal-based metrics, correlational metrics, and classification-based metrics, as shown in Table <ref type="table">3</ref>.</p><p>We conduct our analysis (1) per synthetic data generator, because each has a different hyperparameter space and different sensitivity to changes in hyperparameter configuration;</p><p>(2) per metric, because the best-performing hyperparameter is defined with respect to a specific metric; and (3) per privacy loss budget &#949;. We quantify the degradation in performance when using the synthetic generator on the private data by comparing the best hyperparameter setting that we would have chosen with the private data (i.e., the optimal case) relative to the hyperparameters we would have chosen with each of the (potentially surrogate) public datasets.</p><p>To aggregate the usefulness of public data in choosing hyperparameter configurations across different evaluation metrics, we computed a relative performance degradation metric for each configuration. Concretely, for every private synthetic data generator, privacy level &#949; and dataset (ACS, EDAD, and WE), we first identified the hyperparameter configuration that yielded the best performance on the private reference dataset (i.e. the real data). We then determined, for each candidate surrogate public dataset (and the regular public data), the hyperparameter configuration that would have been chosen based solely on its corresponding performance. Our benchmark quantifies degradation as the relative difference between the performance achieved by the surrogate-chosen hyperparameters on the private reference and the optimal performance on the reference dataset (measured as either absolute error or percent degradation, depending on the metric). We conducted this process independently for each metric -across classification, correlation, and marginal-based metrics. We averaged across multiple experimental seeds to obtain aggregate performance with standard error; we then conducted a Pareto frontier analysis <ref type="bibr">[39]</ref> across the frontier defined by aggregating into the three metric categories: classification, correlation and marginal-based metrics.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.1.3 Task 3: Privacy-Utility Estimation for Synthetic Data</head><p>Understanding the privacy-utility trade-off of a mechanism for a specific private dataset is extremely useful for producing a differentially private release in the real world <ref type="bibr">[94]</ref>. For example, it may provide guidance on setting the privacy loss budget by exposing its impact on the fidelity of private synthetic data (e.g., <ref type="bibr">[4,</ref><ref type="bibr">55]</ref>).</p><p>In this task, we evaluate how well a public dataset -either traditional or surrogate -can estimate the privacy-utility curve for each utility metric. This experiment is, in a sense, the "dual" of the hyperparameter tuning task described in the previous section: here, we compare the privacy-utility curve computed on the public data with the curve obtained on the private data. To mimic real-world usage, we run the DP mechanism with the best-performing hyperparameters determined from the public data (Table <ref type="table">3</ref>), selecting the optimal configuration independently at each tested &#949; value.</p><p>For each dataset, synthetic data generator, and evaluation metric, we created both public-based and private-based curves over a range of privacy loss budgets &#949;. To aggregate the results across different evaluation metrics, we first compute, for each metric group (classification, correlation, and marginals) and each synthesizer (PrivBayes, AIM, and GEM), an aggregated performance value that is the average "chosen value" across all metrics in that group. For a given synthesizer and for each &#949;, we group the results by dataset and reference dataset and then pivot these averages so that each row corresponds to a dataset and each column to an &#949; level. This representation enables us to generate line plots to visually assess the similarity between performance curves (see, e.g., Figure <ref type="figure">28</ref> for an example with the PrivBayes synthesizer).</p><p>Since the line plots alone are insufficient to quantify aggregate closeness, we compute both &#8467; 1 and &#8467; 2 distances between each pair of curves. The &#8467; 1 distance is more interpretable -being in the same units as the evaluation metric -while the &#8467; 2 distance is less sensitive to outliers. We average the &#8467; 1 and &#8467; 2 distances across the different metric categories (weighting each category equally). To reduce variability, each configuration is run 10 times. Finally, we perform a Pareto frontier analysis across both &#8467; 1 and &#8467; 2 distances for each dataset <ref type="bibr">[39]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2 Datasets</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.1 ACS</head><p>The ACS data excerpt was released by the US Census Bureau in September 2020 and provided by the NIST CRC to assess synthetic data generation methods. We designated the "National" dataset (27,254 records) as the private split and the "Massachusetts" dataset (7,634 records) as the public split. Since the differential privacy synthetic data generators assessed in this project are primarily designed for categorical data, we used the "demographic" subset containing 7 categorical features provided by NIST CRC. After removing records with missing values, we retained 23,006 and 6,514 records for the private and public splits, respectively. The public split was up-sampled to match the size of the private split. For a complete description of the dataset and its curation, refer to its documentation <ref type="bibr">[103]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.2 EDAD</head><p>The EDAD (Survey on Disability, Personal Autonomy and Dependency Situations) datasets were released by the Spanish National Statistics Institute (INE) in April 2022 and April 2024, containing responses from their 2020 (164,254 records) and 2023 (12,518 records) surveys respectively. We designated the 2023 survey responses as the private split and the 2020 survey responses as the public split. Since our synthetic data generators are primarily designed for categorical data, we used a subset of 11 categorical features from both surveys. After removing records with missing values, we retained 8,922 and 1,469 records for the private and public splits, respectively. The private split was down-sampled to match the size of the public split. For a complete description of the datasets and their curation, refer to the documentation given by <ref type="bibr">[56]</ref>.</p><p>.2.3 WE The Workplace Equity Survey datasets (WE) consist of responses from two global surveys conducted in 2018 (released December 2019) and 2023 (released April 2024) by the Coalition for Diversity and Inclusion in Scholarly Communications C4DISC). We designated the 2023 survey responses (1,755 records) as the private split and the 2018 survey responses (1,182 records) as the public split.</p><p>Since our synthetic data generators are primarily designed for categorical data, we used a subset of 12 categorical features from both surveys. In this dataset, we kept the missing values as another category. We retained 837 and 1,400 records for the public and private splits, respectively, and no upsampling or downsampling was done. The slight reduction in records is due to filtering response with high levels of missingness and only using respondents from the top 10 most common country affiliations in the survey (to reduce dimensionality). For a complete description of the datasets and their curation, refer to their documentation <ref type="bibr">[98,</ref><ref type="bibr">69]</ref>.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.2.4 Dataset Memorization by the LLMs</head><p>Recent research has highlighted growing concerns that, because LLMs are exposed to benchmark data from the internet during training, their performance those and other benchmarks may be inflated when assessing performance post-training <ref type="bibr">[78,</ref><ref type="bibr">45,</ref><ref type="bibr">91,</ref><ref type="bibr">116,</ref><ref type="bibr">36]</ref>. For example, it is well known that LLMs have a large capacity for training data memorization <ref type="bibr">[24,</ref><ref type="bibr">62,</ref><ref type="bibr">25]</ref>; this is one mechanism by which they could "hack" existing benchmarks, by simply memorizing the examples and their answers. This memorization consideration is particularly relevant for our experimental setup, where we utilize LLMs to generate records both directly and indirectly. Thus, any prior exposure to our evaluation datasets (ACS, EDAD, and WE) could significantly impact model performance in our evaluations (of particular concern is exposure to the split of these datasets that we consider private in our evaluations, e.g., the national version of the ACS dataset). We address this memorization concern through two mitigation strategies.</p><p>First, we considered the temporal relationship between dataset releases and model knowledge cutoff dates when selecting two of our datasets for evaluation. Namely, the private splits of EDAD and WE were released in April 2024, which is later than the knowledge cutoff dates of most models used in our study (Table <ref type="table">2</ref>): GPT-4o (October 2023), Llama 3.3 70B (December 2023), and Claude 3.5 Sonnet (April 2024). While there is a one-month overlap with Claude, the analysis of Cheng et al. <ref type="bibr">[29]</ref> suggests that the effective knowledge cutoff dates of LLMs typically precede their reported dates.</p><p>Second, we executed the LLM memorization assessment methodology proposed by <ref type="bibr">[18]</ref>; they provide an extensive package &amp; benchmark for LLM memorization detection specific to tabular data. We ran their assessment across all private and public splits. In the data generation tests from <ref type="bibr">[18]</ref> the most relevant to our setting -both header tests (generating the first few rows) and row completion tests (generating random-location rows) indicated no evidence of record-level memorization by any of the three LLMs across all datasets. Refer to Figure <ref type="figure">8</ref> for an example of the header test results for the ACS dataset with Claude 3.5 Sonnet.</p><p>Additional tests examining an LLM's metadata knowledge of tabular datasets, rather than record generation capabilities, revealed varying levels of dataset familiarity. The models unsurprisingly demonstrated strong familiarity with ACS, but limited knowledge of EDAD and minimal recognition of WE. This pattern aligns with the relative public visibility of these datasets: ACS is a core and official product of the US Census, EDAD is an official product of the Spanish National Statistics Institute, and WE is a small-scale survey conducted by a coalition of professional and trade organizations.</p><p>When provided with header columns and the first few rows, all models successfully identified the name of the ACS dataset, and sometimes could identify the EDAD dataset name (where the 2020 public split consists of multiple raw files). However, for the WE dataset, even when given headers and first rows, no model generated the correct dataset name instead, they provided thematically related names such as "work-life-and-career-survey" and "publishing-industry-diversity-survey." We hypothesize that this pattern emerges from the survey questions themselves serving as column names, which inherently reveal the overall topic of the survey (e.g., "How long have you worked in publishing and/or related industries?").</p><p>We observed similar patterns regarding column name completion. When given the dataset name and the first few features, all models failed to generate the correct column names for both EDAD and WE datasets. For ACS, the models could generate some of the column names, but not in the correct order. We hypothesize that this is due to the fact that the ACS datasets we used were sub-sampled, modified, and adopted from the US Census release by NIST.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3 Private Mechanisms</head></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3.1 Classification</head><p>Differentially private pretraining is usually conducted in domains where strong, publicly available priors with matching data-dimensionality are available (e.g., text or image data). In these fields, neural transformer models dominate <ref type="bibr">[114,</ref><ref type="bibr">64]</ref>.</p><p>For an adequate analog to this space in the tabular setting, we consider an FTTransformer model <ref type="bibr">[46]</ref>, which is a transformer based architecture for tabular data classification. FTTransformer has demonstrated strong performance against established powerful gradient boosting approaches such Figure <ref type="figure">8</ref>: The header test output on the ACS dataset on Claude 3.5 Sonnet. The LLM is prompted with the column names as well as a few first rows of the dataset (black), and its completion is presented. The output is colored according to its Levenshtein string distance compared to the original records: correct, incorrect, and missing. We observe that the LLM failed to reproduce the header, as many errors occur within columns with variability.</p><p>as XGBoost <ref type="bibr">[28]</ref>. Its effectiveness stems from specialized data transformations that mitigate information loss in transformer-based attention layers <ref type="bibr">[47]</ref>. Prior work shows how simple it can be to adapt FTTransformer to the private setting <ref type="bibr">[95]</ref> by making minor modifications to its architecture to support DP-SGD <ref type="bibr">[1]</ref>. Importantly, it can also be easily pre-trained with public data through standard gradient updates before private training. The differentially private variant of FTTransformer is (&#949;, &#948;)-DP, for which we set &#948; = 10 -5 .</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.3.2 Data Synthesis</head><p>We considered three representative state-of-the-art private data release methods: PrivBayes <ref type="bibr">[120]</ref>, GEM <ref type="bibr">[73]</ref> and AIM <ref type="bibr">[80]</ref>. Each of these synthesizers follows the "Select-Measure-Project" paradigm, in that they privately select statistical queries (marginals or correlations) to run on a sensitive distribution, privately measure these queries, and then as post-processing project these measurements onto a synthetic distribution (from which we can draw arbitrary samples) that approximates the original, sensitive distribution.</p><p>PrivBayes builds a Bayesian network (BN) and adds noise to all k-way correlations to ensure differential privacy. Despite having been published in 2017, PrivBayes is still considered state-of-the-art and was chosen to produce the differentially private release of the Israel National Live Birth Registry <ref type="bibr">[55]</ref>. GEM parameterizes a neural model to represent a synthetic distribution that approximates the true distribution by minimizing a linear query error based loss (with linear queries implemented as k-way marginals, where by default k = 3). AIM relies on the Private-PGM graphical model <ref type="bibr">[79]</ref> to parameterize the underlying distribution, and utilizes an iterative process to take advantage of higher values of &#949;. Both AIM and GEM are considered the state-of-the-art approaches to generating private synthetic data <ref type="bibr">[102,</ref><ref type="bibr">93]</ref>. Outside of these methods, we acknowledge that many other methods exist for generating DP data <ref type="bibr">[37,</ref><ref type="bibr">51,</ref><ref type="bibr">109,</ref><ref type="bibr">81,</ref><ref type="bibr">117,</ref><ref type="bibr">92,</ref><ref type="bibr">10,</ref><ref type="bibr">23]</ref>, but we believe that PrivBayes, GEM and AIM are a representative set of what can be currently considered state-of-the-art.</p><p>PrivBayes and GEM are &#949;-DP, whereas AIM is (&#949;, &#948;)-DP, for which we set &#948; = 10 -9 . All three methods come with hyperparameters that need to be tuned. Detailed lists of hyperparameters persynthetic data generator, and their associated values, are given in Appendix D.4.    Figure 13: Mean AUC Advantage of the DP model after pretraining, grouped by generation method.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>D.4 Hyperparameter Spaces</head><p>The mean is calculated across the DP finetuning hyperparameter space when best pretraining hyperparameter configuration is chosen for the pretraining step, with 10 runs per hyperparameter configuration.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E Additional Result Discussion</head><p>In this section, we present the results of our evaluation framework (Section 4) for the following DP auxiliary tasks: pretraining (Section E.1), hyperparameter tuning (Section E.2), and estimating the privacy-utility trade-off (Section E.4). Appendix F provides additional results details.</p><p>All of the experiments were done with &#949; &#8712; {1, 2, 4, 8, 16}, and each hyperparameter configuration (Appendix D.4) was run 10 times.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E.1 Results for Task 1: Pretraining for DP Classification</head><p>In our analysis, the best pretraining hyperparameter configuration was selected based on the public validation subset (see Figure <ref type="figure">15</ref> and Table <ref type="table">24a</ref> for full hyperparameter averaging results, which show similar trends).</p><p>EDAD and WE. Overall, we find strong evidence that LLM-based methods -both CSV and Agent surrogate public data generation (particularly with Claude 3.5 Sonnet) -offer a competitive alterna- .11 (.77) Agent (Claude 3.5 Sonnet, Max Cov.) .20 (.73)  .17 (.72)  .15 (.73)  .06 (.69)  .07 (.73) Agent (GPT-4o, Unif.) -.01 (.52) .02 (.58) -.01 (.57) -.04 (.59) -.06 (.60) Agent (GPT-4o, Max Cov.) -.05 (.48) -.05 (.50) -.07 (.51) -.05 (.58) -.02 (.63) Agent (Llama 3.3 70B, Unif.) .00 (.54) .03 (.59) .04 (.62) .02 (.66) .02 (.68) Agent (Llama 3.3 70B, Max Cov.) -.01 (.52) -.01 (.54) .02 (.60) .02 (.65) .02 (.68) Agent (All, Unif.) .09 (.62) .15 (.71) .12 (.70) .05 (.68) .07 (.73) Agent (All, Max Cov.) .08 (.61) .14 (.69) .13 (.71) .07 (.71) .06 (.72)</p><p>tive to traditional public data. Figure <ref type="figure">9</ref> presents our experimental results on the WE and EDAD (&#949; = 1), demonstrating how pretraining on the surrogate public data can vastly improve the starting point of model performance.</p><p>Figure <ref type="figure">13</ref> shows a diminishing pretraining advantage when increasing &#949; for both EDAD and WE. This is an expected behavior: high epsilon allows for the extraction of more signal from the private dataset, and may reduce the usefulness of public data, regular or surrogate <ref type="bibr">[104]</ref>. Table 13: Pareto Efficient Methods (Task 2: Hyperparameter tuning for private synthetic data) for AIM on ACS. E.2 Results for Task 2: Hyperparameter Tuning for DP Synthetic Data Generation ACS. On ACS, where LLMs are likely to possess well-calibrated priors due to extensive training on U.S. Census data (see Appendix D.2.4), the AIM synthesizer (Table <ref type="table">13</ref>) shows that Agent (Claude, Unif.) is best for both classification (0.004) and marginal consistency (0.024), while CSV (Claude) has the best correlation metric (0.002) (although the Agent based Claude methods here are close behind). For the PrivBayes synthesizer (Table <ref type="table">10</ref>), the CSV-based approaches are impressive: CSV (GPT) achieves the best in terms of classification metrics (0.001), CSV (Llama) is best in marginal metrics (0.041), and CSV (Claude) is best for correlation metrics (0.033). For GEM on ACS, the Agent (Claude, Max Cov.) approach is dominant along with the Arbitrary baseline. Recall that the Arbitrary baseline directly encodes relationships into the data (via the Bayesian approach described in Appendix C.1.3). In the case of GEM, whether relationships between variables are accurate to true relationships in the private data is less important when tuning its hyperparameters.</p><p>EDAD. We now turn to the EDAD dataset (a Spanish disability survey); EDAD was published after many LLMs training cutoffs, so we expect the LLMs to have less, if any, prior exposure to tabular data in the same domain as the schema we present. For the AIM synthesizer (Table <ref type="table">14</ref>), several agent-based methods (e.g., Agent (All, Unif.)) are similarly strong for classification metrics (0.001). Although the correlation metrics are tightly grouped (ranging from 0.014 to 0.019), the overall Pareto frontier is defined by a mix of the CSV and Agent approaches. For the PrivBayes synthesizer (Table <ref type="table">11</ref>), the agent-based method Agent (Claude, Unif.) again leads on classification (0.004) while CSV (Claude) remains on the Pareto frontier for correlation (0.046); meanwhile, the real Public data yields the best marginal consistency (0.097).</p><p>WE. For WE -the Workplace Equity survey dataset, also from a period after many LLM training cutoffs -for the AIM synthesizer (Table <ref type="table">15</ref>) the best-performing methods are exclusively Agentbased methods. Here, Agent (Claude, Unif.) leads in classification metrics (0.016), Agent (GPT, Unif.) attains the best correlation metrics (0.007), and Agent (GPT, Max Cov.) provides the strongest marginal consistency (0.025). In contrast, for the PrivBayes synthesizer (Table <ref type="table">12</ref>), although the Arbitrary baseline dominates on marginal consistency (0.056) and is competitive on correlation (0.052), agent-based methods (both All, Max Cov. and Claude, Unif.) yield a substantial improvement in classification performance (0.016 -0.019).</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>E.3 DP-GAN Results</head><p>We also tested DP-GAN as a potential DP synthesizer for hyper-parameter tuning. However, the poor performance of the model on the tasks we considered was such that we decided not to pursue additional results or consider it along the mainline, state-of-the-art DP synthetic data methods (which included GEM, a similar neural generative adversarial synthetic data method). We include some limited results here in case they are of interest to the reader, in Tables <ref type="table">19</ref> and <ref type="table">20</ref>.  <ref type="table">23</ref>, the distances between the performance vectorsmeasured in both &#8467; 1 and &#8467; 2 norms -vary considerably across datasets and synthesizers. For example, in the AIM synthesizer (Table <ref type="table">21</ref>), methods such as Agent (All, Max Cov.) achieve an ACS &#8467; 1 of 0.039 and an ACS &#8467; 2 of 0.023, while CSV (Llama) attains similar values (ACS &#8467; 1 : 0.044, ACS &#8467; 2 : 0.023). In the GEM setting (Table <ref type="table">22</ref>), a similar trend is observed. Here, the Arbitrary baseline exhibits impressively low EDAD &#8467; 1 (0.028) and EDAD &#8467; 2 (0.013) distances, while other methods, such as Agent (All, Unif.) and CSV (GPT), also display competitive performance on certain metrics. For the PrivBayes synthesizer (Table <ref type="table">23</ref>), CSV (GPT) achieves an ACS &#8467; 1 of 0.091 and an ACS &#8467; 2 of 0.042 -values that are generally lower than those produced by several agent-based approaches on other metrics.</p><p>Method ACS &#8467;1 EDAD &#8467;1 WE &#8467;1 ACS &#8467;2 EDAD &#8467;2 WE &#8467;2 Arbitrary (Baseline) 0.353 0.510 0.367 0.184 0.231 0.166 Agent (All, Max Cov.) 0.364 0.258 0.330 0.186 0.116 0.148 Agent (All, Unif.) 0.519 0.126 0.373 0.274 0.057 0.168 Agent (Claude, Unif.) 0.705 0.257 0.254 0.355 0.115 0.124 Agent (Llama, Max Cov.) 0.543 0.559 0.260 0.288 0.251 0.119 Agent (Llama, Unif.) 0.337 0.696 0.295 0.176 0.312 0.133 Table 21: Priv/Util Pareto Efficient Methods (Task 3: Privacy/utility tradeoff estimation) for AIM. Method ACS &#8467;1 EDAD &#8467;1 WE &#8467;1 ACS &#8467;2 EDAD &#8467;2 WE &#8467;2 Univariate (Baseline) 0.321 0.028 0.294 0.144 0.013 0.133 CSV (GPT) 0.091 0.155 0.402 0.042 0.070 0.180 Agent (All, Max Cov.) 0.094 0.071 0.318 0.043 0.033 0.144 Agent (All, Unif.) 0.112 0.051 0.280 0.051 0.024 0.126 Agent (GPT, Max Cov.) 0.127 0.061 0.232 0.058 0.027 0.105 Table 23: Priv/Util Pareto Efficient Methods (Task 3: Privacy/utility tradeoff estimation) for PrivBayes.</p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>F Additional Detailed Results</head><p>In this section, we present additional detailed results of our evaluation framework (Section 4 and Appendix D) for the following DP auxiliary tasks: pretraining, hyperparameter tuning, and estimating the privacy-utility trade-off.</p><p>F.    A g e n t (A ll , U n if .) A g e n t (L la m a 3 .  A g e n t (A ll , M a x C o v. )</p><p>A g e n t (A ll , U n if .) A g e n t (L la m a 3 . 0.0 0.5 1.0 1.5 Avg. Abs. Degradation U n iv a ri a te (B a se li n e ) A rb it ra ry D o m a in (B a se li n e ) P u b li c A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) C S V (G P T-4 o ) C S V (C la u d e 3 .5 S o n n e t) C S V (L la m a 3 .3 7 0 B ) A g e n t (C la u d e 3 .5 S o n n e t, U n if .) A g e n t (G P T-4 o , U n if .) A g e n t (G P T-4 o , M a x C o v. ) A g e n t (A ll , M a x C o v. ) A g e n t (A ll , U n if .) P ri v a te Classification 0 2 4 6 Avg. % Degradation U n iv a ri a te (B a se li n e ) D o m a in (B a se li n e ) A rb it ra ry P u b li c A g e n t (A ll , M a x C o v. ) A g e n t (A ll , U n if .) C S V (L la m a 3 .3 7 0 B ) A g e n t (G P T-4 o , M a x C o v. ) C S V (G P T-4 o ) A g e n t (G P T-4 o , U n if .) A g e n t (C la u d e 3 .5 S o n n e t, U n if .) A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) C S V (C la u d e 3 .5 S o n n e t) P ri v a te Correlation 0 2 4 6 Avg. % Degradation A rb it ra ry U n iv a ri a te (B a se li n e ) D o m a in (B a se li n e ) A g e n t (A ll , U n if .) C S V (C la u d e 3 .5 S o n n e t) C S V (G P T-4 o ) P u b li c A g e n t (C la u d e 3 .5 S o n n e t, U n if .) A g e n t (A ll , M a x C o v. ) A g e n t (G P T-4 o , M a x C o v. ) A g e n t (G P T-4 o , U n if .) A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) C S V (L la m a 3 .3 7 0 B ) P ri v a te Marginals Figure 21: Granular hyperparameter tuning results for EDAD on the AIM synthesizer. Several agentbased methods, such as Agent (All, Unif.), deliver strong classification performance (0.001), with the Pareto frontier defined by a mix of CSV and agentbased approaches (correlation metrics ranging from 0.014 to 0.019).  A g e n t (A ll , U n if .) A g e n t (G P T-4 o , M a x C o v. ) A g e n t (L la m a 3 . 0.00 0.25 0.50 0.75 1.00 Avg. Abs. Degradation U n iv a ri a te (B a se li n e ) C S V (C la u d e 3 .5 S o n n e t) A g e n t (G P T-4 o , M a x C o v. ) D o m a in (B a se li n e ) P u b li c A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) A g e n t (C la u d e 3 .5 S o n n e t, U n if .) C S V (L la m a 3 .3 7 0 B ) A g e n t (G P T-4 o , U n if .) A g e n t (A ll , M a x C o v. ) A rb it ra ry A g e n t (A ll , U n if .) C S V (G P T-4 o ) P ri v a te Classification 0 10 20 30 Avg. % Degradation D o m a in (B a se li n e ) C S V (C la u d e 3 .5 S o n n e t) A g e n t (C la u d e 3 .5 S o n n e t, U n if .) A g e n t (A ll , U n if .) A g e n t (G P T-4 o , M a x C o v. ) A g e n t (A ll , M a x C o v. ) P u b li c A g e n t (G P T-4 o , U n if .) A rb it ra ry C S V (L la m a 3 .3 7 0 B ) C S V (G P T-4 o ) U n iv a ri a te (B a se li n e ) A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) P ri v a te Correlation 10 20 30 40 Avg. % Degradation C S V (C la u d e 3 .5 S o n n e t) A g e n t (C la u d e 3 .5 S o n n e t, U n if .) A rb it ra ry A g e n t (G P T-4 o , U n if .) C S V (L la m a 3 .3 7 0 B ) U n iv a ri a te (B a se li n e ) A g e n t (G P T-4 o , M a x C o v. ) A g e n t (A ll , U n if .) A g e n t (A ll , M a x C o v. ) D o m a in (B a se li n e ) C S V (G P T-4 o ) A g e n t (C la u d e 3 .5 S o n n e t, M a x C o v. ) P u b li c P ri v a te Marginals Figure 24: Granular hyperparameter tuning results for EDAD on the GEM synthesizer. As in ACS, both the agentbased approach and the Arbitrary baseline perform competitively.      to capture, we plot the average performance across epsilon that constitutes each vector, relative to the true performance of the sensitive data (which, in these plots, is the black dotted line). Ideally, for privacy/utility estimation, any public data (surrogate or otherwise) would match the performance of the private data across privacy loss budget parameters. This would allow a practitioner to, say, choose the correct based on a performance threshold in absolute terms. Clearly, given the noisiness of the lines (which generally cluster around, but inconsistently track, the black dotted line for private data performance), this is a difficult estimation problem. </p></div>
<div xmlns="http://www.tei-c.org/ns/1.0"><head>G Details of Dataset Similarity G.1 Statistical Distance Metrics</head><p>We now introduce metrics for comparing probability distributions and datasets used throughout this paper.</p><p>Definition 3 (Total Variation Distance). For discrete probability distributions P and Q over X , the Total Variation Distance (TVD) is defined as:</p><p>The Total Variation Similarity (TVS) is simply 1 -TVD(P, Q), representing the similarity rather than the distance between distributions. Both TVD and TVS can be naturally extended to datasets by considering the empirical probability distributions induced by the datasets over the universe X . Now we turn to a more specific measurement of disparity between two datasets based on the results of statistical queries.</p><p>Definition 4 (Linear Query). Given a predicate &#966; : X &#8594; {0, 1} that maps database records to binary values, a linear query q &#966; : X n &#8594; N + 0 is a function that, for a dataset D &#8712; X n , computes:</p><p>In other words, a linear query counts the number of records in dataset D that satisfy the predicate &#966;. Definition 5 (Workload Error). Given a workload W = {q 1 , . . . , q k } of linear queries, and a pair of datasets D, D &#8242; &#8712; X n , the workload error is defined as:</p><p>The average k-way marginal error can be defined as a special case of the workload error where the workload W consists of all possible k-way marginal queries. For instance, the 3-way marginal error uses all possible triplet combinations of attributes as queries. Assuming datasets of equal size, the average k-way marginal error is normalized by both the number of queries in the workload |W | and the size of the datasets |D|: AvgError k-way (D, D &#8242; ) = 1 |W | &#8226; |D| &#8721; q&#8712;W |q(D)q(D &#8242; )| where W is the set of all k-way marginal queries, and |W | = ( d k ) for a dataset with d attributes.</p></div><note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="2" xml:id="foot_0"><p>https://github.com/shlomihod/surrogate-public-data</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="3" xml:id="foot_1"><p>With one exception: the univariate baseline, which samples directly from the sensitive data without correlation between variables. This method is introduced purely for comparison, and is not a valid public data surrogate under our working defition.</p></note>
			<note xmlns="http://www.tei-c.org/ns/1.0" place="foot" n="4" xml:id="foot_2"><p>At worst there is a one-day overlap; in addition, our verbatim-memorization tests<ref type="bibr">[18]</ref> show no evidence of row-level leakage; see Appendix D.2.4.</p></note>
		</body>
		</text>
</TEI>
