The advent of large-scale generative models such as GPT-4 (Achiam et al., 2023), DALL-E (Ramesh et al., 2022) and Stable Diffusion (Rombach et al., 2022) has revolutionized the field of artificial intelligence. These models, trained on vast web-scale datasets, exhibit remarkable capabilities in generating text, images, and other media (Brown et al., 2020;Saharia et al., 2022). However, as these models become more widely used, an increasing amount of generated data populates the web. This raises a critical question: what are the consequences of training generative models on datasets containing their own outputs?
Recent studies have investigated this question, revealing that training generative models on their own outputs can cause the performance of such models to progressively degrade with each model-fitting iteration, eventually rendering newer models useless (Hataya et al., 2023;Martínez et al., 2023a;Shumailov et al., 2023;Alemohammad et al., 2023;Martínez et al., 2023b;Bertrand et al., 2023;Briesch et al., 2023;Dohmatob et al., 2024a;b) (see Appendix A for review and discussion of prior work). This phenomenon was consequently labeled model collapse. Model collapse warns that democratizing access to generative models runs the risk of polluting the very data necessary to train future iterations of generative models.
To better understand this phenomenon many prior works have considered a setup that assumes each model's generated data replaces previous data. In theory, this leads to very natural comparisons across generations as the total number of training points for each model remains fixed. In practice, subsequent generations of LLMs are often trained with increasing data over time -e.g., 1.4 trillion tokens for Llama 1 (Touvron et al., 2023a), 2 trillion for Llama 2 (Touvron et al., 2023b), 15 trillion for Llama 3 -in which presumably both human-generated and machine-generated data are accumulating in training sets collected from the internet. It was noted in some of those works Hataya et al. (2023); Martínez et al. (2023a); Alemohammad et al. (2023); Bertrand et al. (2023); Dohmatob et al. (2024b) that model collapse can be either slowed down or negated by mixing in clean data with the generated data.
To that end, in this work we study the effect of accumulating data on model collapse, rather than replacing data. Our data-accumulating setting is, in some sense, maximally pessimistic: it considers a hypothetical future where synthetic data are uncontrollably dumped on the internet to be vacuumed up for training the next iteration of generative models. Nevertheless, we find that model collapse is avoided when accumulating data.
We begin by studying model collapse experimentally with deep generative models trained on realistic data: transformers on causal language modeling (Sec. 2.1), diffusion models on molecular conformation (Sec. 2.2) and variational autoencoders on images (Sec. 2.3). After confirming that replacing data at every iteration indeed causes test error to increase with the number of iterations, we empirically find that accumulating synthetic data with real data avoids model collapse for all models and for all data modalities we test. To understand why replacing data and accumulating data have different consequences for model collapse, we turn to an analytically tractable framework of a sequence of linear models, each trained on synthetic outputs generated from the previous-iteration's fitted linear model (Mobahi et al., 2020;Dohmatob et al., 2024a). Within this framework, Dohmatob et al. (2024a) demonstrated that if data are replaced with each model-fitting iteration, the test error increases linearly with the number of iterations n. We extend Dohmatob et al. (2024a)'s analysis to prove that if data instead accumulate, then the test error has a finite and (to us, surprisingly) well-controlled upper bound independent of the number of model-fitting iterations. 1Altogether, our work suggests that data accumulation may be robust to model collapse and emphasizes the importance of considering accumulating data and other real-world data dynamics in the analysis of model collapse in generative models trained on web-scale data.
We first investigate model collapse experimentally in several classes of generative models.
Here, and for the remainder of this manuscript, the term model collapse refers to notably worsening error over increasing iterations of the model-data loop, while avoiding model collapse refers instead to bounded error over such iterations. To test the effect of accumulating data on model collapse, we compare accumulating data against replacing data. We use three diverse experimental setups of causal transformers, diffusion models, and variational autoencoders trained on real text, molecular conformation, and image datasets, respectively. We find that replacing data yields model collapse for all models and all datasets, whereas accumulating data avoids model collapse.
Experiments We first train causal transformers (Vaswani et al., 2017) on text data. Specifically, we pretrain 9M parameter GPT-2 (Radford et al., 2019) and 12M, 42M and 125M parameter Llama2 (Touvron et al., 2023b) language models for a single epoch on TinyStories (Eldan & Li, 2023), a 470M token GPT-3.5/4-generated dataset of short stories at a kindergarten reading level. For each model-fitting iteration n ≥ 2, we sample a new dataset of the same size as TinyStories from the previous iteration's language model and then either replace or concatenate the previous dataset with the newly generated dataset.
In each model-fitting iteration, we then pretrain a newly initialized model on the replaced or concatenated dataset from the previous iteration. We experiment with sampling the new datasets using temperatures 0.3 or 1.0. We chose this combination of architectures, scales, dataset, and sampling because the setup necessitates pretraining multiple iterations
1 2 3 4 5 Model-Fitting Iteration 1.75 2.00 2.25 2.50 2.75 Cross Entropy (Test) Replace 1 2 3 4 5 Model-Fitting Iteration Accumulate
Figure 2: Data Accumulation Avoids Model Collapse in Language Modeling. Sequences of causal transformer-based language models are pretrained on TinyStories (Eldan & Li, 2023). Cross-entropy validation loss increases when repeatedly replacing data (left), but not when accumulating data (right). Synthetic data was sampled with temperature = 1.0. of language models -a computationally costly endeavor -but we also wish to study realistic conditions where generative models are high-performing and generative diverse outputs.
Because small language models (below 10M parameters) pretrained on TinyStories were shown to be able to generate coherent-albeit-simple English sentences (Eldan & Li, 2023), this choice of architectures, scales, dataset and temperature hopefully strikes a good balance between being representative, being diverse and being computationally feasible.
We found that for all architectures, parameter counts, and sampling temperatures, as the number of model-fitting iterations increased, replacing data led to an increase in test cross entropy (Fig. 2 top). We also found that for all architectures, parameter counts, and sampling temperatures, as the number of model-fitting iterations increased, accumulating data led to equal-or-lower test cross entropy (Fig. 2 bottom). Lower temperature (0.3) led to a faster increase in test error than higher temperature (1.0) (Appendix Fig. 13), but the trend was consistent for both temperatures. Table 1 shows samples of generated texts for GPT2 (9M) and Llama2 (125M) models at model-fitting iterations 3-5 when both accumulating and replacing data, as well as iterations 8-10 (replacing only).
In the end, the crab found a smooth shell. He took it to a safe place under a tree. The crab put the shell where he found it. Tim and his mom were tired, but they were happy. They had a fun day at the beach. And they lived happily ever after. The end.
Henry asked his Mom why the golf sounded so special. His Mom explained that the line of lumber had something special that would help. She said that if you're not sure, the lumber is special. 8 (R)
Friend Stan and Millie laughed together and prepared to spend the morning together. Mamaing Grandma's possibilitant, twice would measure how much she lovedk. Everyone started to get ready when they started arguing until their mum upset. GPT2 (9M) 5 (A) Jack was so happy that he took care of the honey.
He thought, "I care about the beautiful garden, because it is nice and clean." He started to feed the flower every day. The flower grew bigger and taller, and Jack became very happy. 5 (R) After playing, Lily got tired and quickly ran back to playing with her dolls. She opened her eyes and played with her dolls all day long. Her grandma was so happy that she screamed as she watched her look back at her original clothes and laughed. 10 (R) When she finished eating it, she tasted it all up. She said goodbye to her mom and said goodbye. Mommy smiled, feeling very proud of her. It was other. She knew that sharing is always easy to share her meal with her mom. Table 1: Data Accumulation Avoids Model Collapse in Language Modeling. Both 125Mparameter Llama2 as well as 9M GPT-2 models show decreasing quality when replacing data (R), but maintain high-quality text generations when accumulating data (A).
We ablate for several additional potential confounds beyond generation temperature. First, when accumulating data, subsequent model iterations are trained on larger datasets than when replacing data. To control for this, we also perform experiments in which data is replaced, but the size of the (fully synthetic) dataset is grown to match the training set size in the accumulation regime. We find that model performance still degrades (albeit at a lower rate). This is shown in Appendix C, Table 2, right-most column. Second, a possible concern could be that degrading performance when replacing data could be due to low model performance in iteration 1 (and thus the quality of the first synthetic dataset). We control for this by varying the amount of training performed in iteration 1 only and find that this has no significant impact. Lastly, we find that our results are also consistent across varying dataset sizes and training epochs. These ablations are discussed in Appendix F.
Experiments We next train sequences of diffusion models on molecular conformation data. Specifically, we train GeoDiff (Xu et al., 2022), a geometric diffusion model for molecular conformation generation, on the GEOM-Drugs (Axelrod & Gomez-Bombarelli, 2022) dataset. We down-sample the training split of GEOM-Drugs to 40, 000 molecular conformations, which we use as our initial training set, and perform 50 diffusion steps for each prediction.
For the loss, we use the standard loss used by GeoDiff: a weighted variational lower bound to the conditional likelihood; for more details, see Xu et al. (2022).
Results Over 8 model-fitting iterations, we find test loss increases when replacing data, matching our language model experiments, and test loss remains relatively constant when accumulating data (Fig. 4). Unlike with language models, we found that when replacing data, performance worsens significantly in the first model-fitting iteration trained on synthetic data and does not degrade further substantially in subsequent iterations.
Experiments We lastly train sequences of variational autoencoders (VAEs) (Kingma & Welling, 2013;Rezende et al., 2014) on CelebA (Liu et al., 2015), a dataset of 200k images of human faces split between train and test sets, chosen as a balance between being a realistic dataset with many samples, color images and resolution, and computational feasibility of training multiple iterations of models on accumulating data. The loss is the standard VAE loss: reconstruction error plus the KL divergence between the encoder's output Gaussian and the isotropic Gaussian prior. See Appendix D for more experimental details.
We find that replacing data at each iteration again exhibits model collapse: the test error rises swiftly with each additional iteration, and each iteration yields lower quality and less diverse generated faces until all model generations represent a single mode as shown in the left panel of Figure 6. In contrast, accumulating data at each iteration significantly slows model collapse: the test error increases significantly slower with each additional iteration. While the diversity of generations does go down as compared in the middle and right panel of Fig. 6, it still represents major axes of variation in the dataset, such as gender, but no longer seems to generate other details, along more minor axis of the data manifold, such as glasses and accessories. We discuss further analysis of VAE reconstructions in Appendix D.
Interestingly, unlike language modeling, the test error of accumulating data does increase with the number of iterations (albeit much more slowly than with replacing data). We also note that Martínez et al. (2023a) found slightly contradictory evidence, specifically that a different architecture on a much smaller dataset exhibits fast performance deterioration even with accumulating data. Understanding under what conditions and why these discrepancies exist is an interesting direction we leave for future research.
To gain mathematical understanding and intuition, we employ an analytical framework introduced in prior work (Mobahi et al., 2020;Dohmatob et al., 2024a) to understand the difference between data accumulation and data replacement. We will show that it predicts the same types of test error behaviors for these two data-use strategies that were measured empirically. The framework considers a sequence of linear models that are fit to the synthetic data sampled from the linear generative model model based on the previously fit linear models. Within this framework, Dohmatob et al. (2024a) showed that if data are replaced across model-fitting iterations, then the test squared error increases linearlyfoot_1 with the number of iterations n. Here, we extend Dohmatob et al. (2024a)'s argument to show that if data instead accumulate across model-fitting iterations, then the test squared error is upper bounded by a relatively small constant, meaning model collapse is avoided 3 . The content in this section relies heavily on the framework and pioneering contributions of Dohmatob et al. (2024a). Our contribution is to study a different way to use synthetic data in training, namely accumulate, which seems to better align with certain real-world considerations. We show that our empirical results could have been anticipated on theoretical grounds, by applying the same analysis framework as in Dohmatob et al. (2024a), but instead to this specific training dataset pattern. We use the same framework to analyze some other ways that synthetic data might have be used, such as replace, again the theory aligns with many empirical results.
Original Data Distribution. We adapt notations from Dohmatob et al. (2024a). Define the distribution P Σ,w,σ 2 on R d × R given by (x, y) ∼ P Σ,w,σ 2 iff :
The positive integer d is the input-dimension, the matrix Σ ∈ R d×d is the true covariance structure of the input x, the vector w * is the true linear relationship used to generate the original data and the scalar σ is the level of label noise. We start at iteration n = 1 with T initial independent data points (x i , y i ) each following P Σ,w * ,σ 2 , that is,
T as rows. We also form the vectors Y and E with i-th coordinate y i and ϵ i respectively. In whatever follows, we will assume that X has full column rank, i.e., T ≥ d, X ⊤ X is invertible and the model is underparameterized.
Synthetic Data Generation Process. We generate synthetic data from the following sequence of distributions
where n ∈ N is the number of iterations. The scheme is outlined as follows.
• For n = 1:
Here, for a matrix A with full column rank, A † = (A ⊤ A) -1 A ⊤ is the Moore-Penrose pseudo-inverse of A. The noise terms E 1 , E 2 , . . . , E n are independent of each other and of the covariates/features. Since X has full column rank, so does Xn for every n ≥ 1. use the term MAD to refer to the situation where the distance between the distribution of the original data and that of the subsequent generative models grow farther apart, without necessarily diverging. We consider sequences of linear models recurrently fit to generated targets by previous iterations of models. Top: If each linear model is fit to the generated targets of only the preceding linear model, i.e., data are replaced, then the test error grows linearly with the number of iterations n. Bottom: If each linear model is instead fit to the generate targets of all the preceding linear models, i.e., data accumulate, then the test error has a finite upper bound independent of the number of iterations. This suggests that data accumulation might be a robust solution for mitigating model collapse. For log test error and higher iterations, see Appendix Fig. 16.
Test Error. We are interested in the dynamics of the test error E test ( ŵn ) of this sequence of linear model ŵ1 , ŵ2 , .... Note that evaluation of the model is done on the true distribution P Σ,w * ,σ 2 , even though the model is trained on the accumulated synthetic data. For any linear estimator ŵ computed from the training data, we measure test error in the standard way:
where the expectation is taken over the training data and (x test , y test ) ∼ P Σ,w * ,σ 2 independent of the training data.
To reiterate a comment made previously by Dohmatob et al. (2024a), although we present our results in the context of ordinary linear regression in R d , our analysis can be readily extended to ridge regression and the kernel setting (Caponnetto & De Vito, 2007;Simon et al., 2021;Cui et al., 2021;Wei et al., 2022). We focus here on a simple useful model for studying model collapse.
Our goal is to establish an analytic formula for the test error of the nth model in the data accumulation setting. We begin by characterizing the relationship between the fitted linear parameters ŵn and the true parameters w * . We remind the reader that we assume that X has full column rank, i.e., X ⊤ X is invertible. Proofs are deferred to App. B.
Theorem 1. In the data accumulation setting, ∀n ≥ 1, the fitted linear parameters ŵn can be expressed as:
where, recall, w * is the true parameter, X is the original design matrix, and E i is the extra noise added at the i'th iteration.
Theorem 2. For an n-fold synthetic data generation process with T ≥ d + 2 samples per iteration and isotropic features (Σ def = I d ), the test error for the ridgeless linear predictor ŵn learned on the accumulated data up to iteration n is given by:
where, recall, σ 2 is the noise variance of the fake data generation process, d is the input dimension, and T is the number of samples (i.e., data points) added per iteration.
How does test error with accumulating data compare against test error with replacing data? Under otherwise identical assumptions, Dohmatob et al. (2024a) proved in the datareplacing setting that the test error is given by 4 :
When data are replaced, the test error grows linearly with the number of iterations n (Fig 7 top), with the rate of growth determined by a noise-to-signal ratio: the amount of noise per dimension σ 2 times the number of dimensions d, adjusted by the (per-iteration) sample size T. In contrast, when data accumulate, Theorem 2 shows the test error is upper bounded regardless of the number of iterations n:
This striking difference can be intuitively explained by the differences in the way data are handled across iterations. In the data replacement setting, because previous data were discarded, the model is more strongly affected by the new noise that each iteration of generated data introduces, and adds that to the effects experienced in earlier iterations. But in the data accumulation setting, because iteration i contributes fraction 1/i to the training dataset, the additional noise from the ith iteration of synthetic data has its effect on the model MSE shrunken proportional to 1/i 2 (due to squared error). The summability of 1/i 2 prevents the test error from growing indefinitely. This suggests that accumulating generated data with real data can indeed avoid model collapse.
To confirm the analytical results, we numerically simulate the setup. The numerics almost perfectly matched the analytics (Fig. 7): when data are replaced, the test error grows with the number of iterations n, with the prefactor set by the noise-to-signal ratio σ 2 d/(Td -1), but when data accumulate, the test error rapidly plateaus with the prefactor similarly set. For log test error and higher model-fitting iterations, see Appendix Fig. 16.
This work explored the phenomenon of model collapse, an important concern as AIgenerated content permeates the internet and finds its way into future training datasets.
Prior work has shown that training on model outputs can lead to degraded performance (Martínez et al., 2023a;b;Shumailov et al., 2023;Alemohammad et al., 2023;Hataya et al., 2023;Bertrand et al., 2023;Briesch et al., 2023;Dohmatob et al., 2024a;b), implying that Our findings extend these prior works to show that if data accumulates and models train on a mixture of "real" and synthetic data, model collapse no longer occurs. We show this both experimentally on causal transformers for language modeling, diffusion models for molecule generation, and variational auto-encoders on image data as well as theoretically for linear regression. Together, these results strongly suggest that the "curse of recursion" may not be as dire as had been portrayed -provided we accumulate synthetic data alongside real data, rather than replacing real data by synthetic data only.
Looking to the future, many questions worth investigating remain.
For instance, in future work we would like to explore different data generation and accumulation regimes, such as (1) additional "real" data being introduced in each model-fitting iteration and (2) different schedules of how much synthetic data is generated at each iteration and (3) human-filtering of generated data, e.g., as done in RLHF. Additionally, we note that in all our experiments, the synthetic dataset is generated by sampling from the previous model, i.e., with some stochasticity; in future work, we would like to explore also what happens if data is generated deterministically, e.g. with temperature 0 in a typical language model. Lastly, it is worth noting that "model collapse" -as a term of art -has been used in various ways by various researchers; so care is required in comparing claims across articles. In reviewing the literature, we identified at least four related phenomena: (0) unbounded test error blowup (as here); (1) modal collapse -collapse to one (or a few) modes; (2) collapse to uniformity; and (3) amplification of artifacts introduced by models fit to previous synthetic data. Future work should map out what factors cause which to occur and what preventative strategies are effective at addressing each. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022. A Summarization and Discussion of Prior and Related Work Prior Empirical Work A growing body of recent work has investigated the phenomenon of iteratively training models on data generated by previous models, e.g., Hataya et al. (2023); Martínez et al. (2023a); Shumailov et al. (2023); Alemohammad et al. (2023); Martínez et al. (2023b); Bertrand et al. (2023); Briesch et al. (2023); Dohmatob et al. (2024a;b) and (in a different context) Taori & Hashimoto (2023). Hataya et al. (2023) and Martínez et al. (2023b) conducted experiments replacing real training data with generated data at each iteration, assuming that the dataset size remains fixed over time. They found that this iterative retraining procedure can lead to model degradation if the proportion of synthetic data becomes too high. Similarly, Shumailov et al. (2023) ran experiments with Gaussian mixture models, VAEs, and language models in which the total number of samples per iteration was held constant, and the samples always originated with the previous model rather than aggregating over time. Building on this work, Alemohammad et al. (2023) considered three treatments of data: fully replacing real data with synthetic data, augmenting a fixed real dataset with additional synthetic data, and mixing new real data with synthetic data at each iteration. In almost all of their experiments, they drew a fixed size dataset from the most recent model at each iteration, without accumulating data. Bertrand et al. ( 2023) also assumed that dataset size and mixing proportions are constant over time in their theoretical stability analysis and empirical validation. They are able to conclude that the test error grows linearly with the iteration count in their setup, as well as derive more interesting and more nuanced results using random matrix theory. They also discuss how to mitigate model collapse through optimal regularization both when the training data are noise-free and noisy versions of the previous model's synthetic outputs. A related noise-free setup was studied by Mobahi et al. (2020) in the case of self-distillation. Although Mobahi et al. (2020) considers a more general setup with ridge regression as a special case, they use noiseless predictions from the previous model as the training data for the next model, and show that eventually, the predictions shrink to zero. Through this, they highlight that self-distillation induces regularization in the function space, which initially is beneficial for reducing over-fitting, but eventually overregularization causes underfitting and hence performance decay. Dohmatob et al. (2024b) go beyond the linear model to study model collapse -they study the tails of LLM outputs vs. real data and provide scaling laws that clearly identify regimes of model degradation when synthetic data misses tails present in real data. They identify an interesting phase transition in the test error scaling law depending on the size of the real dataset size in comparison to (a functional of) the chopped-off tail, and conclude that enough real data is able to mitigate model collapse. All these works consider the scenario where the amount of training data available per iteration is fixed (and does not grow with the iteration count), and it is certainly possible that with larger amount of synthetic data (from prediction by the previous model), several of these scalings would improve significantly. For example, in Equation ( 12) of Dohmatob et al. (2024b), one obtains the linear scaling (with iteration count) of test error simply because the amount of synthetic data generated per iteration is the same. If one generated synthetic data with size proportional to the iteration count, then at iteration n, the scaling would, instead of n, be like n 1-c /(1c) for c < 1. When one does not increase the dataset size, Dohmatob et al. (2024b) points out that increasing the proportion of real data would help one to avoid model collapse altogether. However, even if one did increase the amount of synthetic data with iteration count, Theorem 3.2 coupled with Corollary 3.3 in Dohmatob et al. (2024b) would tell us that the amount of real data was all that matteredif the amount of real data is large, we overcome model collapse. If one only had synthetic data (and no real data), no matter how large, it would be impossible to regain the original real-data scaling laws. The scenario we study is highly inspired by these pioneering works, but still, in our view, different. We consider the case when we keep augmenting synthetic data (generated by the previous model trained on all the previous data so far) as iterations progress, much akin to how -in our view -the internet evolves. We observe that we can avoid model collapse in this setting. The analysis of previous models in our case is more involved, since the data used for training at iteration n is not homogeneous -different models from the past impart different statistical aspects to different parts of the training data. We also note a related augmentation model studied by Jain et al. (2024) -they perform risk minimization augmenting real data with synthetic data available from a potentially different independent source. One of their messages is that augmentation of (even) pure noise can be looked upon as adding a ridge penalty and hence, in certain cases, can improve test error. Their setup, however, is different from ours, since the synthetic data in their setup is not obtained by a learning algorithm employed on the real data, and the process is not iterative. However, morally, each iteration of ours involves risk minimization on data statistically composed of an equal mixture of data generated from the previous models, and hence each iteration of ours can be mapped to the general framework developed in Jain et al. (2024), although the dependencies among the various models trained in our setup introduce theoretical complications that do not seem to be too easily addressed by the theory developed in Jain et al. (2024). Shortly after v1 of our manuscript was uploaded to ArXiv, two other manuscripts appeared, dealing with the theoretical aspects in a setting similar to ours. Theorem 1 of Marchi et al. (2024) obtains the same square summability scaling of the variance as us. Seddik et al. ( 2024) studies collapse in language models in both purely synthetic and partly synthetic regimes and obtains deviation bounds as model iterations progress.
The two papers we found that partially considered accumulating data are Martínez et al. (2023a) and Alemohammad et al. (2023). Alemohammad et al. ( 2023) did so in one-half of one experiment: StyleGAN2 trained on FliqrFaces 128×128 (App. Fig. 8). The authors concluded that accumulating data does not avoid model collapse, but merely slows it down. However, we believe that a closer examination of their results (App. Fig. 8) reveals that accumulating data causes the test error to plateau to a relatively low error with increasing numbers of model-fitting iterations. This result would support our conclusion that accumulating data avoids model collapse and does not merely delay it.
The results from Martínez et al. (2023a) are harder to evaluate; model collapse only seems to occur when the amount of synthetic data added per model-fitting iteration is 2× the total amount of accumulated data, and the subsequent work by the authors switched from accumulating data to replacing data (Martínez et al., 2023b). We think understanding what conditions and why these discrepancies exist is an interesting future direction.
Avoiding Model Collapse Several papers present methods for avoiding or slowing model collapse. Bertrand et al. (2023) shows in the replacing data setting that model collapse will not occur if the initial generative models approximate the data distribution well enough and the proportion of real data is sufficiently large with respect to the synthetic data. Dohmatob et al. (2024b) similarly demonstrates that in the replacing data setting, carefully selecting real data to mix with synthetic data can avoid model collapse. Other solutions may also be possible in various models and under various assumptions. To our knowledge, no paper has claimed an "optimal" strategy to avoid model collapse, and neither has ours.
We point out a lemma useful to prove Theorem 2.
Lemma 3. Let T and d be positive integers with T ≥ d + 2, and let X ∈ R T×d be a random matrix with i.i.d. rows from N (0, Σ) with Σ positive definite. Then, X has full rank a.s. Moreover, it holds that:
(5)
Proof. See Dohmatob et al. (2024a).
Assuming Lemma 3 and Theorem 1, we present the proof of Theorem 2.
Proof of Theorem 2. From Theorem 1, we have:
where w * is the true parameter, X is the original data matrix, and E i are the noise terms at each iteration, with E i ∼ N (0, σ 2 I T ). The test error is given by:
where the expectation is taken over all random quantities involved.
Substituting ŵn into the test error expression and using the fact that Σ def = I d , we get:
Using Lemma 3, we have:
Therefore, the test error for ridgeless regression with isotropic features in the data accumulation setting is:
Finally, we prove Theorem 1.
Proof of Theorem 1. We prove this theorem by induction.
Base case: For n = 1, we have:
Inductive step: Assume that for some n ≥ 1, we have:
Recalling that Ŷi :
Substituting this back into the expression for ŵn+1 :
Now, using the induction hypothesis:
Now, we need to simplify the term ∑ n i=1 ∑ i j=1 E j j + ∑ n+1 i=1 E i . We can do this by counting the number of times each E i appears in the double sum: E 1 appears n times in the double sum and once in the single sum, so its coefficient is n+1
1 . E 2 appears n -1 times in the double sum and once in the single sum, so its coefficient is n 2 . This continues along till we reach E n , which appears once in the double sum and once in the single sum, so its coefficient is 2
n . E n+1 appears only once in the single sum, so its coefficient is 1 n+1 . Therefore,
Substituting this back into the expression for ŵn+1 :
Therefore, by mathematical induction, the lemma holds for all n ≥ 1.
Model training was implemented using Huggingface Transformers (Wolf et al., 2019). Dataset generation was implemented using vllm (Kwon et al., 2023).
In addition to Figure 3 in the main text, Figures 9-12 show learning curves in larger print, with x-axes showing either epochs or gradient steps, and with axes shown in linear-linear or log-log scale, respectively.
In addition to the experiments shown in the main paper, we conducted several ablation studies.
Controlling for dataset size. One possible concern is that when accumulating data, the train dataset size will grow at each model-fitting iteration, meaning subsequent models will be trained on more aggregate data than their counterparts in the replacement regime. To control for this, we run experiments controlling for this. In this "replace-multiple" regime, we create a fully synthetic dataset at the end of each model-fitting iteration, but grow the size of this dataset to match that of the accumulated data in the accumulation regime. Table 2 rightmost column shows that in this regime, evaluation loss still increases over model-fitting iterations.
Generation temperature. Most of our language model experiments were run with sampling temperature 1.0 during generation of new datasets. To ensure that this choice is not critical, we also run one experiment with temperature 0.3, and see that this shows similar results (with even larger increases in validation loss in the replacement regime than temperature 1.0), as shown in
Table 2, row 2, and Figure 13. Dataset size and training epochs. We similarly vary the size of the initial (and subsequent) training datasets and number of training epochs, and see that this has no qualitative effect on the results (Table 2, rows 3 & 4 show training on 1/5th of the TinyStories dataset for 1 & 3 epochs, respectively). Model quality after first model-fitting iteration. Finally, we control specifically for model (and thus synthetic dataset) quality after the first iteration, to rule out an undue influence of a "bad" first synthetic dataset on subsequent training. Figure 14 shows performance in subsequent iterations for different amounts of training in the first iteration, showing no qualitative differences. Model t=1 t=4 (acc) t=4 (repl) t=10 (repl) t=4 (*) GPT-2 (9M) 1.82 1.74 (-0.07) 2.39 (+0.58) 2.91 (+1.09) 2.18 (+0.36) GPT-2 (9M) (temp=0.3) 1.82 1.75 (-0.06) 5.82 (+4.00) 9.85 (+8.04) n/a GPT-2 (9M) (small dataset) 2.56 2.28 (-0.28) 3.21 (+0.65) 3.72 (+1.16) 2.91 (+0.35) ibid (+ 3 epochs) 1.99 1.87 (-0.12) 2.62 (+0.63) n/a n/a Llama-2 (12M) 2.06 1.94 (-0.12) 2.72 (+0.66) n/a n/a Llama-2 (42M) 1.90 1.76 (-0.14) 2.52 (+0.62) n/a n/a Llama-2 (126M) 1.71 1.59 (-0.12) 2.23 (+0.53) n/a n/a Table 2: Evaluation cross-entropy loss for different models at model-fitting iterations 1, 4 and 10 for replacement and accumulation regimes. (*) indicates a replacement regime with growing dataset size to ablate for total train set size.
In the framework of Mobahi et al. (2020) and Dohmatob et al. (2024a), we consider sequences of linear models fit to the previous model's synthetic outputs. Within this framework, Dohmatob et al. (2024a) proved that if data are replaced with each model fitting iteration and the training data cardinality remains constant, then the test squared error scales linearly with the number of model fitting iterations n: E Replace test
In this work, we lightly adapt the argument of Dohmatob et al. (2024a) to study the effects if data accumulate with each model fitting iteration. We specifically considered the case where the training data cardinality increases by a constant T with each model-fitting iteration i.e. the ith model is fit using T × i data, where T data are "real" and then each subsequently fit model contributes its own T synthetic data to the accumulating data. In this setting, the test squared error is upper bounded independent of the number of iterations.
In the main text, we focus on the replace and accumulate data settings because prior work focused on replacing data and we wished to study how accumulating data affects model collapse. However, a much richer landscape of outcomes is possible. For instance, and as pointed out in personal correspondence with Dohmatob et al. (2024a), one can consider what we term the "Replace-Multiple" setting, in which one fits the i-th linear model using T × i data sampled from the (i -1)-th linear model. Replace-Multiple is a useful baseline for Accumulate because it matches the amount of training data at each model fitting iteration. Under Replace-Multiple, the test squared error grows logarithmically:
Replace-Multiple has the drawback of not matching the total amount of compute of Accumulate since each iteration of Replace-Multiple draws T × i samples from the most recent model, whereas Accumulate draws T samples from the most recent model. Other baselines are also possible, but we leave these to future work. We focus on accumulating data as we feel real and synthetic data are likely to accumulate in the real world as time progresses.
An approach 'halfway' between the 'replace' and 'accumulate' settings replaces the previous dataset with a pure synthetic dataset of size T × i at the i-th iteration; in other words, the comparison at each generation is now between models trained on the same number of training points. While this halfway setting has a milder sublinear behavior -explicitly, the test MSE scaling is MSE ≍ O(log(n)) -the test error does still diverge with iterations. (See Appendix E for more details.) We thank Elvis Dohmatob, Yunzhen Feng, and Julia Kempe for communicating this observation. We also consider the halfway setting as an ablation in our language modeling experiments, as detailed in Appendix C.
To echo an earlier footnote, an approach 'halfway' between the 'replace' and 'accumulate' approaches would replace the previous dataset with a pure synthetic dataset of size iT at the i-th iteration. Analyzing this goes mostly in parallel, except the 1/i 2 mentioned in running text now becomes 1/i for the 'halfway' approach. Consequently, the MSE scaling becomes MSE ≍ O(log(n)); the 'halfway' approach with pure synthetic data but more of it, again has test error growing unboundedly with iterations. Thanks to Elvis Dohmatob, Yunzhen Feng and Julia Kempe for communicating this observation. See Appendix E for an extended discussion.
In this theoretical section, we identify the term model collapse with the situation where test error diverges to infinity (at any rate) as iterations progress. Other authors may employ similar terminology while identifying it with different properties of test error. For example,Alemohammad et al. (2023)