Introduction

With recent advances in natural language processing (NLP), text-based applications have greatly facilitated our lives. These include AI-assisted medical record summaries (Rumshisky et al., 2016), email and document autocomplete tools (Voytovich & Greenberg, 2022;CNN, 2023), and personalized chatbots (Chew, 2022). However, all these applications (among others) rely on collecting private text data from users to train LLMs, which raises serious privacy concerns as LLMs may memorize and leak sensitive information about users (Carlini et al., 2021;Lukas et al., 2023;Wang et al., 2023). Differentially private synthetic text is a promising and actively studied solution (Putta et al., 2022;Bommasani et al., 2019). It aims to create a new text dataset with similar characteristics to the original private data while ensuring privacy by protecting sensitive information in each sample (known as Differential Privacy (DP) (Dwork et al., 2014)). The DP synthetic text can then be used in developing any downstream NLP system without adding extra privacy risks. It also allows the safe sharing of private data more broadly. For example, hospitals can share their private medical data for research purposes by creating a DP synthetic version of their data.

The state-of-the-art DP synthetic text approach is to finetune pretrained generative language models (LMs) on private data with DP-SGD (a DP variant of SGD (Abadi et al., 2016)) (Yue et al., 2023;Kurakin et al., 2023;Mattern et al., 2022a) (short-handed as DP finetune generator; see Fig. 1). Unlike non-DP ML applications, which have been greatly advanced by powerful LLMs such as GPT-4 (Ope-nAI, 2023b) and LLaMA (Touvron et al., 2023a;b) in a short time after they are released, the state-of-the-art DP synthetic text approaches are unfortunately still based on GPT-2. 1 The reasons are: (1) Many powerful LLMs such as GPT-4, Claude, and Bard are only accessible through APIs. DP finetuning them is not feasible. 2 (2) Even though some LLMs (e.g., LLaMA) are open-source, finetuning them with DP is resource-intensive and non-trivial to implement due to the need to calculate per-sample gradients (see § 2).

DP Finetune Generator

A recent DP synthetic data framework called Private Evolution (PE) (Lin et al., 2024) offers a new opportunity to circumvent these challenges by only requiring API access to foundation models, without needing any model training. The high-level idea is to first draw random samples from a foundation model, and then iteratively improve them by selecting (with DP) the most similar ones to the private dataset and querying foundation models to generate more of such samples. PE shows promising results on images by leveraging pretrained Diffusion Models (Rombach et al., 2022): in certain cases, PE achieves an even better privacy-utility trade-off than DP finetuned generators (Lin et al., 2024).

However, extending PE to text is highly non-trivial. PE requires APIs that generate random samples and variations of a given sample, which need to be redesigned for text. In particular, unlike generating image variants in the continuous pixel space where diversity can be easily manipulated using existing model hyperparameters (e.g., guidance scale in diffusion model (Ho & Salimans, 2021)), texts operate in a discrete space, making it challenging to effectively control the generation diversity. In addition, in contrast to images with fixed dimensionality, text data exhibit varied lengths which adds another layer of complexity. To this end, we propose an augmented PE algorithm (AUG-PE) with new generation and selection techniques that allow us to i) elicit a larger set of more diverse and higher-quality texts from LLMs with appropriate sequence length and ii) effectively select the most relevant texts. Our contributions are:

• We propose AUG-PE for high-quality DP synthetic text generation leveraging API access to powerful LLMs. This includes both a practical instantiation of PE on texts and fundamental algorithmic innovations that may benefit future applications of PE. • We conduct comprehensive evalutions of AUG-PE on Yelp, OpenReview (ICLR 2023), and PubMed (Aug 2023) datasets with various LLMs, including GPT-2-series models, , and open-source LLMs. We show that under the same pretrained LM (GPT-2-series) and privacy budget ✏ = 4, 2, 1, AUG-PE can generate DP synthetic text that achieves comparable or even better performance than finetuning baselines in some cases, in terms of downstream task utility and similarity between synthetic and real samples. Leveraging more powerful LLMs such as GPT-3.5 (where DP finetuning is not applicable) and five open-source LLMs (where DP finetuning is hard to implement), the performance of AUG-PE can be significantly improved. Additionally, AUG-PE can be more computationally efficient than DP finetuning by requiring LLM inference APIs only. • We explore the properties of AUG-PE including its text length distribution, its compatibility with stronger LLMs as data generators and downstream models, and its behaviors under data scaling, to provide insights for future development of PE.

Background

Differential Privacy (DP). (✏, )-DP ensures that the output of a randomized mechanism M is close regardless of whether an individual data record is included in the input or not. Specifically, given any pair of two adjacent datasets D, D 0 (i.e., adding or removing one sample), any possible output set E, it holds that Pr[M(D) 2 E]  e ✏ Pr [M (D 0 ) 2 E] + . Moreover, arbitrary postprocessing of the output of an (✏, )-DP mechanism does not incur additional privacy loss, based on the post-processing property of DP (Dwork et al., 2014).

DP synthetic text. To guarantee DP for private training data, one method involves using DP-SGD (Abadi et al., 2016) during model training for specific NLP tasks (Yu et al., 2022;Li et al., 2021). Alternatively, one can finetune pretrained generative language models, such as GPT-2, with private data using DP-SGD and then generate synthetic text datasets (Putta et al., 2022;Bommasani et al., 2019) (Fig. 1). Such DP synthetic texts can be employed in an arbitrary number of non-privately trained downstream tasks without increasing privacy loss. Studies by Yue et al. (2023); Mattern et al. (2022a); Kurakin et al. (2023) indicate that training downstream models on DP synthetic text yields performance akin to directly training them on real data with DP, highlighting the good quality of synthetic data.

However, given that state-of-the-art LLMs (e.g., GPT-4, Claude, GPT-3.5) do not provide model weights, DP finetuning them is infeasible. Even for open-source LLMs (e.g., LLaMA (Touvron et al., 2023a;b)), it is resource-intenstive to perform finetuning (Malladi et al., 2023). Finetuning with DP-SGD is even harder due to the well-known challenges of per-sample gradient calculations for clipping to guarantee DP. Even with optimization techniques (Malladi et al., 2023;He et al., 2022), DP finetuning is still memory and computationally intensive due to large batch sizes and long training iterations required to reach a good fidelity-privacy trade-off (Anil et al., 2021). Here, we study an API-based method for DP synthetic text generation to overcome these challenges, which only requires model inference and is applicable no matter whether the LLM is open-sourced or not.

Additionally, there is a line of work on text-to-text privatization techniques, which provide different privacy guarantees than DP, such as word-level metric DP or sample-level local DP. We defer more discussion and comparison to App. C.11.

Method

Preliminaries on Private Evolution (PE)

PE is recently proposed as an alternative to DP finetuning for DP synthetic data generation (Lin et al., 2024) by merely requiring APIs of pretrained models, and thus is easier to implement and deploy and can leverage API-based models. The original PE algorithm (for unconditional generation) 3 is the L = 1 case in Alg. 1. PE works by first calling RAN-DOM_API that generates random samples from the foundation model (Line 2), and then iteratively: (1) using private samples to vote for their nearest synthetic samples (under embedding model ) to construct a DP_NN_HISTOGRAM (Line 11), (2) drawing samples according to the histogram (Line 15), and (3) passing those samples through VARIA-TION_API which generates new samples that are similar to the given one (Line 16), e.g., images with a similar object.

3 The conditional version of PE is running Alg. 1 for the private samples from each class/label separately; see Lin et al. (2024).

Write a review for restaurants:

A great spot for pizza.

Nice steak, not too pricey.

The pizza was AMAZING!!! Quick pizza delivery. Very good A great spot for pizza.

Nice steak, not too pricey.

A great spot for pizza.

A great spot for pizza.

Please rephrase the review:

A pizza lover's place.

A pizza paradise.

Step 1: RANDOM_API

Step 2.1: DP_NN_HISTOGRAM

Step 2. Step 1 (RANDOM_API, Line 2): we use prompts to generate random samples from the LLM.

Step 2: we iteratively go through steps 2.1-2.3 to refine the synthetic samples towards the private samples.

Step 2.1 (Line 11): each private sample votes for their closet synthetic sample (using self-embedding Line 6 or mean embedding Line 9) in the embedding space induced by embedding model . "A great spot for pizza" gets 2 votes, and the other sample gets 0 votes. We then add Gaussian noise to the votes to ensure DP. This gives us the DP Nearest Neighbor Histogram (DP_NN_HISTOGRAM).

Step 2.2: we resample the generated texts according to the histogram. We assume that only "A great spot for pizza" remains.

Step 2.3 (VARIATION_API): we use prompts to ask the LLM to generate new similar samples, which are the initial synthetic samples in the next iteration. The prompts are simplified for illustration; see App. B for the complete prompts.

While the PE framework is general across modalities, its core components including (the embedding model), RAN-DOM_API (API for generating random samples from the pretrained model), and VARIATION_API (API for generating new samples that are similar to the given one) require domain-specific designs, and the original paper (Lin et al., 2024) only explores their implementation for images. Compared to images, text introduces unique challenges. For example, unlike images which have a fixed dimensionality, the length of text can vary. In addition, the original PE algorithm yields unsatisfactory text quality. In the following, we explore our design choices for each component and propose our augmented version on text, AUG-PE (shown in Alg. 1 and Fig. 2) with new algorithmic techniques to increase the diversity and quality of text generation.

AUG-PE Design

RANDOM_API.

Given the strong instruction-following capability of LLMs, we consider directly using prompts to generate samples (step 1 in Fig. 2). Following Yue et al. (2023), we assume that class labels are non-private. Therefore, we put class label in the prompt (e.g., "restaurant" in Fig. 2). To encourage diverse generation, we propose a pseudo-class approach, where we generate a list of subcategories for each class from GPT-3.5 and randomly sample one subcategory as the keyword to put in the prompt for each generation (e.g., Steakhouse, Bistros for restaurants).

VARIATION_API takes a sample as input and outputs its variations. 4 Unlike image diffusion models used in Lin et al. (2024), text models usually do not provide off-the-shelf variation APIs. Again, we leverage the instruction-following capability of LLMs to implement this via prompting. We propose two variation methods: paraphrasing and fill-inthe-blanks. For paraphrasing, we use the prompt "Please rephrase the below sentences: {input}". For fill-in-theblanks, we mask p% tokens of input as blanks, resulting in masked_input, and use "Please fill in the blanks for the below sentences: {masked_input}" as the prompt. Given the in-context learning ability of recent LLMs, we provide few-shot demonstrations to improve the generation quality.

To add diversity to the generated variations, we create tone candidates (e.g., "in a creative way", "in a professional style"), randomly subsample one tone, and add such phrase into the prompt for each generation.

Algorithm 1 Augmented Private Evolution (AUG-PE) Input: private dataset Spri, noise multiplier , text embedding model , number of synthetic samples Nsyn, K, L Output: Synthetic text dataset Ssyn T Epri = (Spri) S0 RANDOM_API (Nsyn ⇤L )

histogram calculation Histogram t DP_NN_HISTOGRAM(Et, Epri, ) Pt Histogram t / sum (Histogram t ) // synthetic sample selection and generation if L == 1 then S 0 t draw Nsyn samples with replacement from St with probability Pt St+1 VARIATION_API (S 0 t ) save dataset Ssyn t+1 St+1 else if L > 1 then S 0 t rank samples by probabilities Pt and draw top Nsyn samples save dataset Ssyn t+1 S 0 t S j t+1 VARIATION_API(S 0 t ) for j = 1, 2 . . . , L 1 St+1 [S 1 t+1 , ..., S L 1 t+1 , S 0 t ] return Ssyn T Procedure DP_NN_HISTOGRAM(Esyn, Epri, ) Input: synthetic embedding set Esyn = {ej } n j=1 , private embedding set Epri, noise level , distance function d(•, •) Histogram [0, ..., 0] for epri 2 Epri do i = arg min j2[n] d(epri, ej ); Histogram[i] Histogram[i] + 1 Histogram Histogram + N (0, 2 In) return Histogram

Adaptive text lengths in VARIATION_API. The distribution of text length in real-world datasets is usually fat-tailed: most samples are short while a few are long (Fig. 4). In DP-finetuning-based approaches, to faithfully capture long texts, we need to set a large max token length (denoted by max_token). However, this would significantly increase the computation cost. Prior work (Yue et al., 2023) circumvents this problem by setting a small max_token at the cost of the capability to generate long texts. AUG-PE faces the same challenge. Since APIs usually charge by token usage, a high max_token raises costs (as generated text can exceed needs), while a low max_token sacrifices fidelity.

To address the challenge, we leverage PE to learn text lengths automatically by adjusting per-sample max_token adaptively. Specifically, in VARIATION_API, we add "with {targeted_word} words" in the prompt to specify the desired word count in the generation. targeted_word is modified by setting targeted_word = max{original_word + N (0, 2 word ), min _word} where original_word is the word count of input, 2 word is Gaussian noise variance and min _word is a minimal targeted word ensuring useful generations. We set max_token = btargeted_word ⇤ w2t_ratioc for LLM API calls where w2t_ratio is the approximate number of tokens per word (OpenAI, 2023c).

Embeddings calculation and DP nearest neighbor histogram. We use off-the-shelf text embedding models to calculate the embedding of private/synthetic samples. Notably, the embedding of synthetic samples can be defined either by their self-embedding (when K = 0) or the averaged embedding from K variations (when K > 0). After calculating embeddings, each private sample votes for its nearest synthetic sample in the embedding distance, which results in the Histogram t for synthetic samples. As the voting utilizes private samples, we add Gaussian noise N (0, 2

) to each bin of Histogram t to ensure DP. Sample selection and generation. AUG-PE introduces significant enhancements over the original PE for generating more diverse samples and selecting/retaining highquality samples. Specifically, to enhance sample diversity, we propose the following methods: (1) The random sampling based on the histogram probability P t (Line 15) in original PE results in repeated samples, causing performance degradation for S 0 t . To mitigate this, AUG-PE ranks synthetic samples according to their probability and selects only the top N syn samples, enhancing the diversity without sample redundancy (Line 19). (2) Instead of a single variation, AUG-PE generates L 1 variations for each selected sample in S 0 t , creating a larger and more diverse synthetic dataset S t+1 for subsequent iterations (Line 21). (3) We modify the size of the initial dataset to be L times larger than N syn , matching the expanded size of S t+1 (Line 2). To select/retain high-quality samples, we propose the following methods: (1) The selected samples S 0 t are also included in the next iteration's dataset S t+1 , increasing the likelihood of retaining high-quality synthetic candidates (Line 22). ( 2)

For LLMs, we find that when the variation API produces samples with large variations, the averaged embedding from the variations is not representative of the actual sample. Therefore, we use K = 0 so the nearest neighbor voting is performed on the self-embedding of synthetic samples and we directly use those selected, good samples as algorithm's output S syn t+1 S 0 t (Line 20). In practice, we use {K = #variations, L = 1} as original PE, and {K = 0, L = #variations + 1} as AUG-PE, so that the number of API calls for generating variations (i.e., #variations) are kept the same for fair comparisons.

These enhancements position AUG-PE as a more effective method to generate diverse and high-quality synthetic text.

Privacy analysis of AUG-PE follows original PE and we provide detailed privacy analysis in App. A. Specifically, since each private sample only contributes 1 vote for one bin in the histogram (i.e., nearest synthetic sample), the sensitivity is 1. The histograms are privatized by adding Gaussian noise. The adaptive DP composition theorem (Dong et al., 2019) is applied to track the privacy loss across T iterations.

Experiments

Datasets. We evaluate AUG-PE on three datasets: Yelp Review (Inc, 2023), OpenReview, and PubMed abstracts. We use Yelp, a public benchmark providing reviews on businesses, following the choice in prior work for DP synthetic text (Yu et al., 2022). To mitigate the concerns that existing benchmarks are potentially used at LLM's pretraining stage, we crawl the latest reviews for ICLR 2023 submissions from OpenReview websitefoot_4 to construct a new dataset, where the reviews are made public after recent LLMs are trained. We also use PubMed with abstracts of medical papersfoot_5 crawled by Yu et al. (2023) from 2023/08/01 to 2023/08/07 after recent LLMs are trained. Notably, texts from Yelp are mainly in styles of daily conversation, while the other two datasets require domain-specific knowledge about machine learning or biomedical literature when generating DP synthetic replicas. For conditional generation, we use below attributes as labels: the review ratings and business category for Yelp, and the review recommendation and area for OpenReview. For PubMed, we use unconditional generation.

Models.

For data generators, we use GPT-2 (Radford et al., 2019), GPT-2-Medium, GPT-2-Large, GPT-3.5 (OpenAI, 2022), and non-GPT based LLMs including four 7b-sized models -OPT (Zhang et al., 2022), Vicuna (Zheng et al., 2023), Falcon (Almazrouei et al., 2023), LLaMA-2 -as well as one Mixture-of-Expert model Mixtral-8x7B (Mis-tralAI, 2022). For embedding models, we use sentence-transformer (Reimers & Gurevych, 2019). We study more types of embedding models as ablation study in § 4.2.

Metrics. We evaluate synthetic texts regarding (i) accuracy on downstream tasks, and (ii) similarity between real and synthetic data. Downstream tasks: we finetune downstream models on the synthetic text and evaluate their accuracy on the real test dataset. We pick two representative use cases: using DP synthetic text to train DP text classifiers (Yue et al., 2023) and to train efficient DP lanaguge models (Yu et al., 2023). Specifically, we finetune RoBERTa-base (Liu et al., 2019) as text classifiers to classify review ratings and business categories for Yelp, and to classify review recommendations and areas for OpenReview. For PubMed, we finetune BERT Mini /BERT Small (Turc et al., 2019) 7 on synthetic text and evaluate their next-word prediction accuracy. We study more types downstream models as ablation study in § 4.2.

Similarity between real and synthetic data: we quantitively compare (a) embedding distribution distance (i.e., Fréchet Inception Distance (FID) (Heusel et al., 2017), Precision, Recall, F1 score (Kynkäänniemi et al., 2019), MAUVE score (Pillutla et al., 2021), KL and TV divergences (Chung et al., 1989)) and qualitatively compare (b) text length distribution difference (Yue et al., 2023).

Baselines. We consider two SOTA baselines involving DP finetuning: (1) DP-FT-DOWNSTREAM (Yu et al., 2022;Li et al., 2022): finetuning downstream model on real data with DP-SGD. Note that this baseline is not a competitor to our method, since our goal is to generate DP synthetic data and not merely train a downstream model. (2) DP-FT-GENERATOR (Yue et al., 2023): finetuning generator (e.g., GPT-2) with DP-SGD (note that we cannot finetune closed-source GPT-3.5) and using synthetic texts to finetune downstream model with non-private SGD.

We defer more details about the setups, hyperparameters and metrics to App. B.

Understanding the Performance of AUG-PE

Here, we analyze the performance of AUG-PE by answering five research questions about its utility, efficiency, and robustness against empirical privacy attacks under DP compared to DP-finetuning-based baselines.

RQ1: Can DP synthetic texts generated from AUG-PE outperform those from DP-FT-GENERATOR? DP synthetic texts from AUG-PE can have comparable privacyutility trade-off to those from DP-FT-GENERATOR using the same generator, while outperforming it using the Table 1: Evaluation on downstream model accuracy of three methods along 4 data generators. The highest accuracy across all methods ( obtained by AUG-PE ) is bolded (underlined). (i) Compared to DP-FT-GENERATOR, in some cases, downstream accuracy of AUG-PE is higher ( " ) under the same size of GPT-2-series data generator. Leveraging the inherent knowledge within stronger LLM, GPT-3.5, AUG-PE can achieve higher accuracy, especially on challenging datasets OpenReview and PubMed, outperforming DP-FT-GENERATOR by a notable margin. (ii) Compared to traditional method DP-FT-DOWNSTREAM, AUG-PE can also obtain higher accuracy under DP.

1 Yelp Rating Category Rating Category Rating Category Rating Category DP-FT-DOWNSTREAM Original (1939290 / full data) -76.0 81.6 67.5 72.8 67.2 72.0 66.8 71.8 Original (5000) 70.5 75.1 44.8 61.8 44.8 61.8 44.8 61.8 DP-FT-GENERATOR Synthetic (5000) GPT-2 70.3 75.9 68.2 74.1 67.2 73.1 66.4 73.9 GPT-2-Medium 70.0 75.0 69.0 74.6 67.8 74.3 67.4 74.1 GPT-2-Large 70.4 75.4 68.7 74.2 69.8 75.1 68.7 74.6 GPT-2 67.5 74.8 66.4 74.9 " 67.1 74.7 " 66.9 " 74.4 " GPT-2-Medium 67.5 74.9 66.8 74.6 67.7 74.7 " 67.3 74.6 " GPT-2-Large 67.5 74.5 67.3 74.4 " 65.8 74.1 66.5 75.0 " AUG-PE Synthetic (5000) GPT-3.5 68.4 74.1 68.1 74.0 67.8 74.3 67.9 74.0 OpenReview Area Rating Area Rating Area Rating Area Rating DP-FT-DOWNSTREAM Original (8396 / full data) -65.1 50.8 30.5 32.0 30.5 32.0 30.5 32.0 Original (2000) 55.3 47.8 30.5 32.0 30.4 25.5 6.3 19.8 DP-FT-GENERATOR Synthetic (2000) GPT-2 47.5 32.0 32.1 32.0 31.9 32.0 32.1 32.0 GPT-2-Medium 49.7 36.5 40.3 32.0 33.5 31.9 35.5 31.9 GPT-2-Large 48.3 42.9 38.9 33.7 40.4 33.6 38.6 32.1 GPT-2 42.4 32.1 " 39.9 " 32.1 " 38.8 " 32.1 " 37.6 " 32.0 GPT-2-Medium 41.0 32.3 36.9 32.0 36.0 " 32.0 " 36.6 " 32.1 " GPT-2-Large 42.1 32.1 38.8 32.0 38.4 32.0 38.1 32.0 AUG-PE Synthetic (2000) GPT-3.5 45.4 43.5 43.5 44.6 42.8 44.5 41.9 43.1 PubMed BERTMini BERTSmall BERTMini BERTSmall BERTMini BERTSmall BERTMini BERTSmall DP-FT-DOWNSTREAM Original (75316 / full data) -43.5 47.6 30.7 34.1 28.9 32.5 26.7 30.4 Original (2000) 33.5 34.6 2.2 1.1 1.8 0.8 1.4 0.6 DP-FT-GENERATOR Synthetic (2000) GPT-2 30.2 32.4 27.8 29.7 27.6 29.3 27.2 29.2 GPT-2-Medium 31.0 33.1 28.4 30.2 28.1 30.0 27.8 29.8 GPT-2-Large 31.0 33.1 29.2 31.2 29.2 31.1 28.9 31.1 GPT-2 24.5 26.7 24.7 27.0 24.7 26.9 24.3 26.5 GPT-2-Medium 25.5 27.7 25.4 27.6 25.1 27.4 24.9 27.0 GPT-2-Large 25.7 28.0 25.8 27.9 25.5 27.7 25.1 27.2 AUG-PE Synthetic (2000) GPT-3.5 30.4 32.7 30.3 32.5 30.2 32.5 30.1 32.4

stronger generator GPT-3.5. The downstream model accuracy of different methods along 4 generators on different benchmark datasets is shown in Tb. 1.

(1) When using the same LM (GPT-2-series) as the generator for fair comparisons, DP synthetic texts from AUG-PE demonstrate competitive or even better (" ) utility than DP-FT-GENERATOR on Yelp and OpenReview. However, AUG-PE underperforms DP-FT-GENERATOR on PubMed. This is expected because AUG-PE relies on the knowledge within LLMs to generate high-quality texts without domain-specific finetuning, while GPT-2-series models might have limited exposure to biomedical literature (Radford et al., 2019).

(2) AUG-PE only requires API access, making it possible to use closedsource LLM such as GPT-3.5 for generating DP synthetic text. The results of GPT-3.5 outperform not only AUG-PE GPT-2-series, but also DP-FT-GENERATOR GPT-2-series by a significant margin, especially on challenging datasets such as OpenReview and PubMed. It shows that AUG-PE can effectively leverage the inherent knowledge (e.g., medical knowledge, sentiment of reviews, research areas about machine learning) in stronger LLMs to generate higherquality DP synthetic texts.

(3) In addition to downstream utility, we measure the embedding distribution distance between real and synthetic samples. The results in App. C.9 show that AUG-PE can obtain similar and even lower distances (reflected by FID, TV divergence, Recall, F1, and MAUVE scores, etc.) compared to DP-FT-GENERATOR.

(4) Some methods consistently show a 32.0 accuracy for Rating and 30.5 for Area classification, due to the failure of the downstream RoBERTa-base model under DP, always outputting majority class (see App. B for label distributions).

RQ2: Can DP synthetic texts from AUG-PE be a better choice than DP-FT-DOWNSTREAM on real data with DP? AUG-PE obtains comparable and higher accuracy than DP-FT-DOWNSTREAM under DP. (1) Tb. 1 shows that under ✏ = 2, 1 on PubMed, AUG-PE GPT-3.5 with a smaller synthetic dataset size (2k) is sufficient to produce better downstream models compared to models directly trained with DP on the original data of the full (75k) or same size (2k). Similar conclusions hold for other two datasets, and the advantages of AUG-PE on OpenReview are evident across all generators.

(2) DP-FT-DOWNSTREAM performs fairly poor when the data size is small (e.g., 2k on PubMed and OpenReview), indicating that LMs finetuned with DP-SGD is unable to learn meaningful information under DP noises when samples are limited (Yu et al., 2021;Li et al., 2022;Bu et al., 2022). In contrast, postprocessing property of DP allows us to train downstream tasks on DP synthetic text (with any size) via normal training techniques, without incurring additional privacy loss, potentially leading to a better downstream model than DP-FT-DOWNSTREAM. Figure 6: Fill-in-the-blanks with a larger mask probability p% for GPT-3.5 leads to more diverse generation and higher utility on OpenReview.

RQ 3: How does AUG-PE perform across different privacy budget ✏? (1)

Tb. 1 shows that AUG-PE in general achieves better performance as ✏ increases from 1, 2, 4 to 1, suggesting that AUG-PE scales well with the privacy budget ✏. (2) On OpenReview, from ✏ = 1 ! 1, the rating classification accuracy obtained from DP-FT-GENERATOR GPT-2-Large generated text drops from 48.3 ! 38.6, and DP-FT-DOWNSTREAM on full training data drops from 65.1 ! 30.5, while the accuracy of AUG-PE GPT-3.5 exhibits marginal drop 45.4 ! 43.1. It suggests that in some cases, the performance of AUG-PE (paired with powerful generator) can be more robust under DP noise than FT baselines. The reason could be that LMs are vulnerable to the perturbations introduced in model parameters through DP-SGD, whereas AUG-PE strategically adds noise to the histogram votes, effectively preserving the utility. RQ 4: Compared to DP-FT-GENERATOR, how efficient the API-access-based AUG-PE is in terms of GPU hours? With inference API access, AUG-PE is more efficient than DP-FT-GENERATOR that requires DP-SGD finetuning. (1) As shown in Fig. 3, to generate 100k synthetic samples on Yelp under ✏ = 1, given the same generator GPT-2-Large, AUG-PE L = 7 provides 12.7x speedup and L = 2 further provides 65.7x speedup. (2) The running time of AUG-PE is mainly scaled with # API calls, which is associated with the number of variations L 1 in Line 21. (3) The bottleneck of DP-FT-GENERATOR is DP-SGD finetuning: it takes 1764 GPU hours on 32G NVIDIA V100 to finetune GPT-2-Large on Yelp and 7 hours to generate 100k samples, while AUG-PE L = 2 (L = 7) only requires 27 hours (139 hours). It highlights the computational expense of DP-SGD training, particularly for training LLMs, and underscores the efficiency of the API-based DP algorithm AUG-PE. A detailed breakdown of the GPU hours for each setting is in Appendix Tb. 23. (4) We use half precision (FP16) for LLM inference in AUG-PE. With the emerging efficient inference techniques (e.g., Liu et al. (2023)), AUG-PE runtime can be further optimized. RQ 5: How robust AUG-PE is under empircal privacy attacks compared to DP-finetuning-based baselines? We perform state-of-the-art text membership inference at-tacks (MIAs) against the finetuned downstream models on PubMed dataset. We consider three types of MIAs and report the AUC score: (1) PPL thresholds perplexity to predict membership (Carlini et al., 2021); (2) REFER computes the ratio of the log perplexity of the tested model against a reference model (Carlini et al., 2021); ( 3) LIRA uses the ratio of likelihood (Carlini et al., 2022) and we use the pre-trained model as a reference following (Mattern et al., 2023). The results in Tb. 3 show that AUG-PE generally exhibits lower AUC scores under MIAs compared to DP-FT-GENERATOR and DP-FT-DOWNSTREAM. This indicates a higher robustness to empirical privacy attacks, potentially due to the synthetic nature of the data used for downstream model finetuning, which inherently reduces the risk of overfitting to real private data. We defer the details to App. C.1.

Understanding the Properties of AUG-PE

Here we study properties of AUG-PE including text lengths, its compatibility with stronger data generators and downstream models, and its behaviors under data scaling.

RQ 6: Can AUG-PE produce sentence length distributions similar to real data? AUG-PE produces favorable text length distributions. From Fig. 4, we see that the text length distribution of synthetic samples produced from GPT-3.5 through AUG-PE is close to the distribution of the original Yelp data, highlighting the effectiveness of our adaptive sequence length mechanism ( § 3.2). Note that the finetuning baseline requires a fixed max_token (e.g., 128 for GPT-2), which leads to a hard threshold for maximal text length, which is not the case in our method with our adaptive length technique. Nevertheless, there is a peak near 30 tokens for AUG-PE, which is due to the min _word set in the prompt to prevent empty generation. We defer the convergence of text length distributions over PE iterations to App. C.2.

RQ 7: Can AUG-PE benefit from more powerful LLMs? AUG-PE is effective across a wide range of APIaccessible LLMs. We have observed from Tb. 1 that GPT-3.5 can lead to higher downstream accuracy than GPT-2series, especially on PubMed and OpenReview. Here we evaluate more API-accessible, non-GPT based LLMs. (1)

Table 2: Using powerful LLMs as data generators leads to improved downstream accuracy on three datasets.

Yelp

OpenReview PubMed As shown in Tb. 2, under ✏ = 1, 1, those modern LLMs can obtain comparable and even higher accuracy than GPT-3.5 on Yelp, suggesting that AUG-PE can effectively elicit and select high-quality synthetic text from various types of LLMs. Note that DP finetuning often needs to be implemented case-by-case for LLMs and currently lacks opensource implementations for these LLMs, whereas AUG-PE can easily leverage them.

(2) The results on OpenReview and PubMed in Tb. 2 show that GPT-3.5 leads to higher utility than opensource LLMs (e.g. LLaMA-2), demonstrating the stronger generation power of GPT-3.5 in academic/medical domains. Interestingly, Mixtral-8x7B can also generate high-quality synthetic texts for OpenReview, but not for PubMed.

RQ 8: Can more powerful downstream models benefit from synthetic text generated via AUG-PE? The high-quality synthetic text from AUG-PE is better utilized by larger downstream models. (1) From each row in Tb. 4, we see that next-word prediction accuracy monotonically increases with the use of larger downstream models trained on PubMed synthetic text.

(2) Under both ✏ = 1, 1, the smallest model BERT Tiny favors the synthetic texts from DP-FT-GENERATOR GPT-2-Large, while larger models such as LLaMA-2 favor synthetic text from AUG-PE GPT-3.5. This observation underscores the importance of choosing downstream models of a suitable size; employing overly small models could under-estimate the quality of synthetic texts produced by AUG-PE with GPT-3.5. We hypothesize that this is because i) GPT-3.5 generated texts might already be of higher quality in terms of vocabulary, syntax, semantic coherence, etc., compared to generated texts from finetuned GPT-2-Large; and ii) larger downstream LMs like LLaMA-2 can better understand and utilize the nuances in synthetic texts for improved performance than BERT Tiny .

RQ 9: Can we further improve downstream task accuracy with more synthetic samples generated from AUG-PE? To study the scaling law of AUG-PE, we use GPT-2series models to generate {5k,10k,100k} samples for Yelp, and {2k,3k,5k} samples for other two datasets. As shown in App. C.10, under ✏ = 1, 2, 4, 1, AUG-PE in general achieves better performance across all datasets as the data size increases, suggesting that AUG-PE scales well with the number of synthetic samples.

Validating the Design of AUG-PE

As AUG-PE introduces novel sample selection and generation techniques, here we study algorithm components related to the two steps, respectively (under ✏ = 1), and compare its performance against the original PE.

RQ 10: Can AUG-PE surpass original PE? Tb. 5 shows that AUG-PE achieves notable improvement over PE for GPT-2, e.g., +22.6% on Yelp rating classification. We observe similar conclusions for GPT-3.5 in Tb. 24 in App. i) Usage of private data. Tb. 6 shows that the initial samples (generated from Random API) or their variants (generated from Random API + Variation API) exhibit limited utility without using private data. However, the quality of the synthetic text improves notably after just one iteration of AUG-PE (t = 1) when guided by private data, and this improvement continues to amplify with T iterations. We report the results under DP in App. C.8. iii) Embedding models. Tb. 7 shows that larger embedding models such as "sentence-t5-xl" can more accurately capture the nuances of texts in the embedding space, leading to higher utility for GPT-2 generated texts. i) Variation API prompt designs. We evaluate the impact of four types of Variation API prompts on Yelp: paraphrasing and fill-in-the-blanks prompts under zero-shot and fewshot settings.

(1) Qualitatively, we observed that GPT-2 struggles to adhere to the fill-in-the-blanks instruction, often leaving blanks ("__") in the generated texts. In contrast, GPT-3.5 can effectively fill in the blanks, potentially because GPT-3.5 has been instruction-tuned (Wei et al., 2021) and thus follows the instructions better.

(2) The quantitative results in Appendix Tb. 25 reveal that paraphrasing can be an effective strategy for GPT-2, while fill-in-the-blanks yields better results for GPT-3.5.

(3) Fill-in-the-blanks offers more control over the diversity of generated content. By increasing the mask probability p%, we can create more room for imaginative responses from GPT-3.5, leading to more diverse generations. As indicated in Fig. 6, a higher mask probability corresponds to increased accuracy in downstream area classification tasks when using GPT-3.5. iii) Increasing the number of variations L 1 generally enhances performance of AUG-PE as shown in Tb. 9, due to the expansion of the candidate synthetic sample pool, which increases the likelihood of getting high-quality texts. However, generating more variations requires additional API calls, leading to increased computational costs as discussed in Fig. 3. To balance the trade-off between utility and efficiency, we use L = 7 for GPT-2-series experiments.

AUG-PE convergence. We provide generation results showing the convergence of AUG-PE under one private sample in App. C.12, which demonstrate our sample selection and generation process in a more direct manner.

Conclusion

In this work, we propose AUG-PE for DP synthetic text generation without model training. We conduct comprehensive experiments on three datasets and show that AUG-PE can generate high-quality DP synthetic text with comparable privacy-utility tradeoff to DP finetuning baselines under the same data generator. Leveraging more powerful opensource LLMs or API-based LLMs as data generators, AUG-PE can generate DP synthetic text with improved utility.

A. Privacy Analysis

We first introduce a related theorem from Balle & Wang (2018) in Thm. 1. Theorem 1 (Analytic Gaussian Mechanism (Balle & Wang, 2018)). Let f : X ! R d be a function with global L 2 sensitivity . For any " 0 and 2 [0, 1], the Gaussian output perturbation mechanism M

Next, we provide the privacy guarantee for Alg. 1 in Thm. 2 Theorem 2 (Privacy Guarantee for Alg. 1). Let Alg. 1 run T iterations, with noise multiplier (noise is added to each bin of the histogram), the DP mechanism satisfies (", )-DP if and only if p T

2

Proof Sketch. The proof is very similar to the one in Lin et al. (2024). So we just describe the key steps at a high level.

The L 2 sensitivity of the histogram created in each iteration of Alg. 1 is = 1, to which we add Gaussian noise of scale . Therefore T iterations of the algorithm can be seen as the adaptive composition of T Gaussian mechanisms with L 2 sensitivity 1 and noise scale . The privacy loss of the composition is equivalent to that of a single Gaussian mechanism with L 2 sensitivity 1 and noise scale / p T according to the adaptive composition theorem of Gaussian mechanisms (Corollary 3. of (Dong et al., 2019)). Therefore the privacy gaurantee follows from Theorem 1. We evaluate AUG-PE on there datasets:

B. Additional Experimental Details

• Yelp: Yelp data is a public benchmark providing reviews on businesses, and we used the preprocessed Yelp from (Yue et al., 2023). The number of train/val/test samples and label information in Tb. 10. • OpenReview: For OpenReview ICLR2023 data, we crawl the meta-data for each review using the OpenReview Python library,foot_7 and concatenate the fields "summary_of_the_paper", "strength_and_weaknesses" and "summary_of_the_review" as one sample in our dataset. We group the two attributes -review area and recommendation -together as a combination, and drop the training samples from combinations that contain fewer than 50 training samples. The number of samples after such preprocessing and label information is provided in Tb. 10. The number of samples for each class is provided in Tb. 11 and Tb. 12. • PubMed: we use PubMed with abstracts of medical papersfoot_8 crawled by Yu et al. (2023) from 2023/08/01 to 2023/08/07.

The number of train/val/test samples are reported in Tb. 10.

For Yelp and OpenReview, we focus on conditional generation and use two attributes (i.e., labels) for each dataset: the review ratings (ranging from 1 star to 5 stars) and business category for Yelp data, and the review recommendation (ranging from "1: strong reject" to "8: accept, good paper") and review area for OpenReview ICLR2023 data. We then use those labels for downstream classification tasks based on synthetic texts.

For PubMed, we focus on unconditional generation and use next-word prediction as downstream tasks. This is motivated by (Yu et al., 2023)

B.2.1. MODEL AND HYPERPARAMETERS

We consider four LLMs as data generators in AUG-PE via API-access: GPT-2 (Radford et al., 2019), GPT-2-Medium, GPT-2-Large, and GPT-3.5 ("gpt-35-turbo" hosted on Microsft Azurefoot_9 ) (OpenAI, 2022). We provide the default hyper-parameter setup for GPT-3.5 in Tb. 13 and GPT-2 series models in Tb. 14.

The embedding model in AUG-PE is instantiated by the sentence-transformer from HuggingFace. We use "stsbrobertabase-v2" for OpenReview and Yelp and "sentence-t5-base" for PubMed.

After generating the synthetic samples, we remove those with fewer than 100/50 tokens for OpenReview/PubMed. We noticed that samples with token lengths below those thresholds usually result from an unsuccessful API call for paper review/medical abstract generation (e.g. GPT-3.5 refuses to answer).

In terms of downstream models,

• For Yelp and OpenReview, we finetune the pre-trained RoBERTa-base model for all downstream text classification tasks. We set the max sequence length as 512, the batch size as 64, the learning rate as 3e-5, and the number of epochs as 5 for Yelp and 10 for OpenReview.

• For PubMed, we leverage pre-trained BERT Mini and BERT Small released by (Turc et al., 2019), which are lightweight to meet the inference time and computational cost requirements in many real-use cases. These models employ WordPiece tokenization and were trained on Wikipedia and BookCorpus using masked language modeling. During our downstream task fine-tuning, we implement a causal language modeling mask, restricting each token to attend only to its preceding tokens (Yu et al., 2023). We set the max sequence length as 512, batch size as 32, learning rate as 3e-4, the weight decay as 0.01. We finetune 20 epochs for BERT Mini and 10 for BERT Small epochs. • For Yelp data, we generate 100 subcategories under each business category via ChatGPT and use them as keywords in the prompts. • For OpenReview data, we do not generate subcategories, as the review area label (e.g., "Social Aspects of Machine Learning (eg, AI safety, fairness, privacy, interpretability, human-AI interaction, ethics)") already provides detailed information about the area. Instead, we generate a list of writers with their corresponing tones via ChatGPT (e.g., "Postdoctoral Researcher: Advanced and knowledgeable insights", "AI Policy Maker: Concerned with regulatory and policy implications", "Robotics Engineer: Focus on practical applications in robotics") and use them as keywords in the prompt. • For PubMed data, we also generate a list of writers for medical abstracts via ChatGPT, such as "Clinical Researcher, Principal Investigator, Biomedical Engineer", etc., and use them as keywords in the prompt.

We provide the prompts of RANDOM_API for all datasets in Tb. 15 for GPT-3.5 and Tb. 16 for other LLMs.

In terms of VARIATION_API, (1) for GPT-3.5, we utilize fill-in-the-blanks with adaptive text lengths, providing few-shot demonstrations. To obtain {masked_input} used for fill-in-the-blanks, we calculate the tokens for {input} based on GPT-3.5 tokenizerfoot_10 , mask p% of them as blanks "_", and decode them back to the text. (2) In contrast, for GPT-2-series models, we opt for zero-shot paraphrasing with fixed max _token as VARIATION_API. This choice is based on our observation that GPT-2-series models do not follow the instructions of fill-in-the-blanks and adaptive text lengths well, as they are only pretrained on next-word-prediction tasks without further instruction tuning or reinforcement learning from human feedback (RLHF) (Lambert et al., 2022) for blank filling tasks. Moreover, GPT-2-series models do not gain much from few-shot demonstrations for paraphrasing, possibly due to their inferior instruction-following and in-context learning capabilities compared to GPT-3.5.

We provide the prompts of VARIATION_API for GPT-2-series models in Tb. 17 and for GPT-3.5 in Tb. 18,Tb. 19 and Tb. 20. Table 18: Prompts as VARIATION_API for GPT-3.5 on Yelp.

Speaker Prompt

System

You are a helpful, pattern-following assistant. Following Yue et al. (2023), we set = 1 Npriv•log(Npriv) for (✏, )-DP. As different datasets have different sizes of private training data, they require different . We run 10 PE iterations under DP on all datasets. To achieve ✏ = {1, 2, 4, 1}, we use noise multiplier = {15.34, 8.03, 4.24, 0} for Yelp; = {11.60, 6.22, 3.38, 0} for OpenReview; = {13.26, 7.01, 3.75, 0} for PubMed.

User

Table 19: Prompts as VARIATION_API for GPT-3.5 on OpenReview.

Speaker Prompt

System

You are an AI assistant that helps people find information.

User

Based on the area and final recommendation of a research paper, you are required to fill in the blanks for the input sentences {in a selected_tone}. If there is no blanks, please output the original input sentences.

Area: Applications (eg, speech processing, computer vision, NLP) | Recommendation: 3: reject, not good enough Input: __ proposes an__ method_ ROI detection__arial_f_ without attention_. The_ map can_ used____ for__ and____ show_ improvements on different medical__._Strength__ \n-The idea using__actual images_ sali__ generation_ interesting.\n\n_The improvement____aks is significant. \n\nWeak____The___ and_____ experiments are needed_ such as__f___the_ method_ interesting_ but_ novelty_ limited Fill-in-Blanks and your answer MUST be exactly 85 words: This paper proposes an attention generation method for ROI detection by adversarial counterfactual without attention label. The attention map can be used to highlight useful information for disease classification and detection. The experiments show its improvements on different medical imaging tasks. \nStrengths: \n-The idea using counterfactual images for saliency map generation is interesting.\n\n-The improvement for medical imaging taks is significant. \n\nWeaknesses:\n\n-The novelty is simple and limited. \n\n-More experiments are needed, such as existing counterfactual generation.\nthe proposed method is interesting, but the novelty is limited.

Area: {label_1} | Recommendation: {label_2} Input: {masked_input} Fill-in-Blanks and your answer MUST be exactly {targeted_word} words:

Table 20: Prompts as VARIATION_API for GPT-3.5 on PubMed.

Speaker Prompt

System

Please act as a sentence generator for the medical domain. Generated sentences should mimic the style of PubMed journal articles, using a variety of sentence structures.

User

You are required to fill in the blanks with more details for the input medical abstract {in a selected_tone}. If there is no blanks, please output the original medical abstract.

Please fill in the blanks in the following sentences to write an abstract of a medical research paper: {masked_input} and your answer MUST be exactly {targeted_word} words.

B.3. Implementation Details of Baselines.

For DP-FT-GENERATOR, we finetune the GPT-2-series models following the hyperparameters setup in Table 8 of (Yue et al., 2023).

For DP-FT-DOWNSTREAM, we report the hyperparameters for OpenReview and Yelp in Tb. 22,and PubMed in Tb. 21. For a target ✏, a noise multiplier is set as the smallest value such that DP-SGD can run the target number of steps.

Here we provide more details about the metrics regarding embedding distribution distance. We use sentence-transformer "stsb-roberta-base-v2" from HuggingFacefoot_11 to embed the real and synthetic datasets, and use seven evaluation metrics to measure embedding distribution distance: 1) Fréchet Inception Distance (FID) evaluates the feature-wise mean and

C.3. Efficiency in Terms of GPU Hours

In Tb. 23, we provide a detailed breakdown of the GPU hours shown in Fig. 3. We consider the process of generating DP synthetic data given a private dataset. DP-FT-GENERATOR (Yue et al., 2023) requires two steps: (1) finetuning a pretrained data generator with DP-SGD, and (2) generating samples from the finetuned data generator, whereas AUG-PE requires only one step (Alg. 1). In Tb. 23, we list the GPU hours of each step of each method. For Yue et al. (2023), we use the hyper-parameters in their Table 8.

We can see that the majority of the time spent by DP-FT-GENERATOR is the DP-fine-tuning stage, which is already much more costly than the total cost of AUG-PE. This results from two factors: (1) Training is costly due to the backpropagation, especially for large models;

(2) DP-SGD requires per-sample gradients, which further increases the memory and computation cost. In contrast, AUG-PE only requires model inference and does not require model training, and is thus more efficient.

It is also worth noting that once the model is DP finetuned, DP-FT-GENERATOR can efficiently generate many samples with only model inference. It is illustrated by the small GPU hours in the "Generation" step of DP-FT-GENERATOR. In contrast, in AUG-PE, the required GPU hour is positively correlated with the number of samples. Therefore, DP-FT-GENERATOR can become more efficient than AUG-PE when the number of generated samples is large enough. However, the original PE paper (Lin et al., 2024) proposed an efficient way to generate more DP samples after PE is done, by passing the generated samples through VARIATION_API. In the context of text generation with LLMs, this approach is expected to have a similar overhead as generating more samples from the DP-finetuned generator in DP-FT-GENERATOR. We defer the study of this approach to future work.

C.4. Comparison Between AUG-PE and PE

We compare AUG-PE against PE when using GPT-3.5 as the generator on three datasets. The results in Tb. 24 show that AUG-PE is always better than PE on PubMed for GPT-3.5. Moreover, AUG-PE is better for OpenReview Rating classification task and Yelp Rating classification task. As AUG-PE supports PE as a special case by changing the hyperparameters of L and K, the practitioner can adjust those hyperparameters for a specific downstream task and find the best settings to generate synthetic data.

C.5. Ablation Study on Variation API Prompt Design

The results in Tb. 25 show that fill-in-the-blanks prompt (with few-shot demonstrations) yields better results for GPT-3.5. For GPT-2, paraphrasing can be an effective strategy. Although fill-in-blanks leads to high accuracy on Yelp Category classification task, we find that the generated texts have many unfilled blanks "__" upon inspection.

Table 30: Classification accuracy of downstream RoBERTa-base model under ✏ = 1, 4, 2, 1 on Yelp for two downstream tasks: review rating and business category classification. (i) Compared to DP-FT-GENERATOR, in some cases, downstream accuracy of AUG-PE is higher ( " ) under the same synthetic data size and the same GPT-2-series data generator. Leveraging the inherent knowledge within stronger LLM, GPT-3.5, AUG-PE can achieve higher accuracy. (ii) Compared to traditional method DP-FT-DOWNSTREAM, AUG-PE can also obtain higher accuracy under DP with the same synthetic data size.

Data Type (Size) Method Data Generator ✏ = 1 ✏ = 4 ✏ = 2 ✏ = 1 Rating Category Rating Category Rating Category Rating Category Original (1,939,290) DP-FT-DOWNSTREAM -76.0 81.6 67.5 72.8 67.2 72.0 66.8 71.8 Original (100,000) DP-FT-DOWNSTREAM -72.7 75.5 65.0 71.2 64.1 70.0 62.9 68.7 Original (10,000) DP-FT-DOWNSTREAM -70.9 76.2 44.8 61.8 44.8 61.8 44.8 61.8 Original (5,000) DP-FT-DOWNSTREAM -70.5 75.1 44.8 61.8 44.8 61.8 44.8 61.8 Synthetic (5000) DP-FT-GENERATOR GPT-2 70.3 75.9 68.2 74.1 67.2 73.1 66.4 73.9 Synthetic (10000) DP-FT-GENERATOR GPT-2 71.1 75.8 68.2 73.0 67.7 73.2 66.7 73.7 Synthetic (100000) DP-FT-GENERATOR GPT-2 71.0 75.6 66.8 72.6 67.0 72.3 65.5 71.8 Synthetic (5000) AUG-PE GPT-2 67.5 74.8 66.4 74.9 " 67.1 74.7 " 66.9 " 74.4 " Synthetic (10000) AUG-PE GPT-2 67.2 75.1 66.6 75.3 " 66.2 74.9 " 66.0 74.6 " Synthetic (100000) AUG-PE GPT-2 67.1 76.0 " 66.3 75.1 " 66.1 75.0 " 65.7 " 74.5 " Synthetic (5000) DP-FT-GENERATOR GPT-2-Medium 70.0 75.0 69.1 74.6 67.8 74.3 67.4 74.1 Synthetic (10000) DP-FT-GENERATOR GPT-2-Medium 70.7 75.6 68.8 74.4 68.2 73.8 67.5 73.9 Synthetic (100000) DP-FT-GENERATOR GPT-2-Medium 71.9 76.3 68.1 73.9 67.8 74.3 67.9 73.3 Synthetic (5000) AUG-PE GPT-2-Medium 67.5 74.9 66.8 74.6 67.8 74.7 " 67.4 74.6 " Synthetic (10000) AUG-PE GPT-2-Medium 67.5 74.9 67.4 74.9 " 67.6 75.1 " 67.1 74.7 " Synthetic (100000) AUG-PE GPT-2-Medium 68.2 75.8 67.4 75.5 " 66.6 75.3 " 66.2 74.7 " Synthetic (5000) DP-FT-GENERATOR GPT-2-Large 70.4 75.4 68.7 74.2 69.8 75.1 68.7 74.6 Synthetic (10000) DP-FT-GENERATOR GPT-2-Large 70.7 74.3 69.2 74.9 69.7 75.2 68.9 74.6 Synthetic (100000) DP-FT-GENERATOR GPT-2-Large 71.8 74.1 69.5 74.5 68.7 74.5 69.6 74.4 Synthetic (5000) AUG-PE GPT-2-Large 67.5 74.5 67.3 74.4 " 65.8 74.1 66.6 75.0 " Synthetic (10000) AUG-PE GPT-2-Large 67.1 74.7 " 67.1 74.9 66.6 74.7 67.0 74.4 Synthetic (100000) AUG-PE GPT-2-Large 67.3 75.8 " 67.6 75.7 " 66.8 75.4 " 66.0 75.3 " Synthetic (5000) AUG-PE GPT-3.5 68.4 74.1 68.1 74.0 67.8 74.3 67.9 74.0

more synthetic samples lead to better downstream classification accuracy for the three GPT-2-series models on PubMed.

C.11. Comparision to Text-to-Text Privatization Approaches

This is an active line of research on text-to-text privatization techniques for generating differentially private text. We do not directly compare these methods in our main paper due to the key distinctions in privacy definitions:

1. Different privacy definitions. Our method adopts the standard (✏, )-DP defined over neighboring datasets. This contrasts with (a) Word-level metric DP (Feyisetan et al., 2020;Carvalho et al., 2023): a specific metric for measuring word distance needs to be written in the privacy notation, and privacy guarantee is defined over neighboring words; (b) Local DP (Mattern et al., 2022b;Utpala et al., 2023): privacy guarantee is defined over neighboring samples.

Absence of privacy budgets:

The absence of detailed reporting on exact privacy budgets in (Utpala et al., 2023) hinders direct comparisons with our work.

A qualitative comparison between text-to-text privatization methods and our method is shown in Tb. 34.

Next, we compare AUG-PE with the text-to-text privatization frameworks in detail: word-level metric-DP frameworks (Feyisetan et al., 2020;Carvalho et al., 2023) and sample-level local-DP frameworks (Mattern et al., 2022b;Utpala et al., 2023).

Table 31: Classification accuracy of downstream RoBERTa-base model under ✏ = 1, 4, 2, 1 on OpenReview for two downstream tasks: review area and rating classification. (i) Compared to DP-FT-GENERATOR, in some cases, downstream accuracy of AUG-PE is higher ( " ) under the same synthetic data size and the same GPT-2-series data generator. Leveraging the inherent knowledge within stronger LLM, GPT-3.5, AUG-PE can achieve higher accuracy. (ii) Compared to traditional method DP-FT-DOWNSTREAM, AUG-PE can also obtain higher accuracy under DP with the same synthetic data size. It is worth noting that the mechanism for Local DP (taking a sample as input) and the mechanism for DP (taking a dataset as input) are not directly compatible. To establish a fair comparison between the Local DP in (Mattern et al., 2022b;Utpala et al., 2023) and DP employed by AUG-PE, we leveraged the conversion methodology in Feldman et al. (2022) to convert (✏ 0 )-LDP mechanism to (✏, )-DP mechanism for ✏ ⌧ ✏ 0 , which requires shuffling the LDP outputs from each sample. We use the code implementation provided by Feldman et al. (2022). 14 Due to the constraint that ✏ ⌧ ✏ 0 ,foot_14 the maximal Local DP ✏ 0 that can be used for a valid conversion on Yelp (with 1.9M private samples) is ✏ 0 = 8.785, which corresponds to DP ✏ = 1.10.

According to the Local DP guarantee in (Mattern et al., 2022b;Utpala et al., 2023), ✏ 0 = 2 ⇤ n_tokens ⇤ (b 2 b 1 )/temperature, where b 2 , b 1 is the upper/lower bound for each token logit. We set b 2 = 1 and b 1 = 0 following (Mattern et al., 2022b). With temperature = 2, ✏ 0 = 8.785 only allows generating n_tokens = 8 tokens, which significantly hurts the utility of generated texts. To generate n_tokens = 64 tokens for Yelp, one would need at least LDP ✏ 0 = 64 under temperature = 2, and LDP ✏ 0 = 128 under temperature = 1, which far exceeds practical limits for meaningful privacy guarantees. This contrasts with AUG-PE's capability to generate over n_tokens = 1000 tokens while maintaining high quality under tight DP budgets (e.g., ✏ = 1) in our experiments.

Furthermore, our attempt to directly evaluate Paraphraser (Mattern et al., 2022b) and DP Prompt (Utpala et al., 2023) was hindered by several practical challenges.

1. Paraphraser: The dataset used for finetuning in Paraphraser is not publicly available, and the implementation details necessary for replicating the exact privacy guarantees are absent.

2. DP Prompt does not specify the exact ✏ 0 used in the paper, focusing instead on empirical privacy attack accuracy as a proxy. The epsilon values are reported for all other baselines but not for Paraphraser and DP Prompt in Section 4.1 In this retrospective study, we aimed to investigate the prevalence of stroke and identify the factors associated with its occurrence. Data were extracted from medical records, along with symptoms, electrocardiograms (ECGs), and syncope in a cohort of patients with a mean age of 71 years. Of the total 345 patients, 28% had cardiac abnormalities as revealed by ECGs, significantly higher than those without [p<0.001].. (omitted) ones that closely align with the private example.

Table 36: Generation results from AUG-PE under one Yelp private sample. The synthetic sample generated from GPT-3.5 aligns more closely with the training sample (Yelp review about Mexican food) over iterations. The text length of the synthetic sample is controlled through our adaptive length mechanism. Keywords with semantic similarity to the private sample are highlighted . Private Real deal Mexican food in Chadds Ford, PA. Excellent burritos, tacos, and tamales. I plan to be a regular. Meals served traditionally with sliced radishes, pumpkin seeds and lime. Freshly prepared fillings, onion, cilantro, and red and green sauces. Brightly colored building across from Bavarian Motors. Nothing fancy, but great authentic food. iter 0

I recently visited this restaurant having heard that they had one of the best taco stands in town. I was not disappointed! The tacos were incredibly delicious and tasted like they were made with fresh ingredients. The portions were generous, and the prices were extremely reasonable. The service was excellent too, with the staff being attentive and accommodating. The ambience of the restaurant was also great, making it a perfect spot for a casual night out. Overall, I had a fantastic experience and would definitely recommend this place to all those who love tacos. A definite 5-star rating! iter 1

Despite having heard that they had one of the best taco stands in town, I was not disappointed. The tacos were incredibly delicious and tasted like they were made with fresh ingredients. The portions were generous, and the prices were reasonable. The staff was attentive and accommodating.

The ambience was great, making it a perfect spot for a casual night out. Overall, I had a fantastic experience and would highly recommend this place to all those craving Mexican cuisine . A solid 5-star rating from me! iter 2

Despite having heard that they had some of the best Mexican food in town, the crowded parking lot stands in contrast. However, that was not the case. The food tasted authentic and fresh, with the guacamole and salsa being particularly noteworthy. The portions were generous, and the prices were reasonable. The staff was attentive and accommodating. The ambience was great, making it a perfect spot for a date night or a family dinner.

Overall, I had a fantastic experience and would highly recommend this iter 3

Many had heard that they had amazing Mexican food in town, but the crowded parking lot stands in contrast. However, once you step inside, the experience is incredible. The food is authentic, and the guacamole and salsa being particularly noteworthy. Prices were as well quite reasonable.

The staff was attentive and accommodating. The ambiance was great, making it perfect for a date night or family dinner. I had a fantastic experience and highly recommend the restaurant to anyone seeking quality Mexican cuisine. iter 4

Many locals had heard that they had the best Mexican food in town and the hype stands in its truth. However, upon stepping into the experience, it was incredible. The food was fresh, flavorful, and authentic with the guacamole and salsa being particularly noteworthy. The portions were well-sized and satisfying. The staff was attentive and accommodating. The ambiance was cozy and intimate, making it perfect for a romantic night out or casual dinner with friends. I had a great time and highly iter 5

Many locals had raved about the best authentic Mexican food in town and they were not exaggerating. Walking in to the restaurant was refreshing. The food was fresh, the guacamole and salsa were noteworthy. The margaritas were strong and satisfying. The staff were attentive and accommodating. The ambiance was cozy and intimate, making it perfect for a romantic dinner with loved ones. The prices were great and the portions were generous. The fajitas were sizzling and the tacos were packed with flavor. Overall, this iter 6

Many locals have raved about the best authentic Mexican food in town and they were not exaggerating. Walking in the restaurant was refreshing.

The decor was colorful, lively and festive. The margarita was perfectly balanced and satisfying. The waiter was attentive and accommodating. The ambiance was cozy and intimate, making it perfect for a romantic dinner. The chips and salsa were packed with flavor. Overall, the food was great and the portions were generous. The fajitas were sizzling and the chicken was tender. The guacam iter 7

Many locals consistently rave about the amazing authentic Mexican cuisine and we were not disappointed. The flavors were refreshing, the decor colorful and lively, and the margaritas satisfying. The waiter was attentive and accommodating. The outdoor seating was perfect for a romantic dinner and the chips and salsa were packed with flavor. Overall, the food was top-notch and the portions were generous. The chicken enchiladas and guacamole were particularly noteworthy. It was a great experience and we highly recommend this restaurant to anyone looking for a delicious meal and a iter 8

Many foodies rave about the amazing authentic Mexican cuisine and they were not wrong. The flavors were richly robust, colorful and enticing, and the margaritas were top-notch. The enchiladas and guacamole were particularly outstanding. The service was attentive and accommodating and the outdoor seating was perfect for a leisurely dinner and people-watching. From the chips and salsa to the flavorful entrees, the food was superb with generous portions. The menu offers a large selection of delectable options which makes it easy to recommend to anyone looking for a satisfying meal iter 9

Many foodies have raved about the amazing authentic Mexican cuisine. The flavors were bold and delicious. The tacos were rich and flavorful, with enticing toppings and freshly made guacamole. The service was attentive and accommodating, and the outdoor seating was perfect for a warm summer evening. From the crispy chips and flavorful salsa to the perfectly spiced dishes, the food was generous and satisfying. The menu offers a large selection of vegetarian and meat options which makes it easy to please anyone's taste buds. Overall, the experience was excellent and highly recommended for people looking for an authentic Mexican dining experience.