26. Synthetic data and simulation

Data that were never observed

Data sources and collection raised an idea that at the time looked like a detail: that a sample is not obtained only from real data, but can come from a simulation or from synthetic data, something useful for experimentation. Since then, that idea has moved to the center. Generative models can produce tables, texts and images that imitate real ones with a degree of plausibility that was unthinkable a few years ago.

This chapter is about what synthetic data are, what they are for and, above all, what they cannot do. Because the ease of generating them brings a dangerous temptation: treating them as if they were observations of the world.

What a synthetic data point is

A synthetic data point is one generated by a model rather than observed directly. That definition covers a wide range of techniques, some of which have already appeared in this book:

  • Simulation from known distributions. Drawing random values from a normal or binomial distribution, like those covered in Applied probability, is the oldest and most transparent way to produce synthetic data. It is known exactly which process they come from.
  • Resampling. The bootstrap generates new samples from an existing sample by drawing observations with replacement. It does not invent new values, but it does create new combinations.
  • Statistical models fitted to real data. The joint distribution of the variables (including their relationships) is estimated, and new records are then generated from it.
  • Generative AI models. Language or image models that produce examples from instructions, such as fictitious surveys, customer conversations or product descriptions.

What all these techniques have in common is that a synthetic data point inherits the properties of the model that generated it, with its strengths and its limitations.

What they are for

Synthetic data have legitimate and valuable uses:

  • Testing processes. Before running an analysis on real data, it is useful to check that the code works on data with the same structure: the same columns, the same types, rare cases included.
  • Sharing without exposing. When the original data contain personal information, a synthetic version that preserves their statistical properties can let other people work with them. The chapter Privacy, bias and responsibility returns to this point, with its nuances.
  • Exploring scenarios. Simulation makes it possible to answer "what if...?" questions: how a result would change if demand grew, or what sample size would be needed to detect a given difference with a hypothesis test.
  • Teaching and learning. A simulated dataset, whose generating process is known, is ideal for checking whether a method recovers the parameters that were used to create it.
  • Balancing rare classes. In classification problems where one of the categories is very rare, generating synthetic examples of that category can help train a model, taking care not to evaluate it on those same examples.

What they cannot do

The fundamental limitation is simple to state: a synthetic data point contains no information about the world that was not already in the model that generated it. If the model was fitted to a sample of a thousand records, generating a million synthetic records does not produce a million observations: it produces a thousand observations restated many times.

This has a direct consequence for inference. Point and interval estimation shows that the larger the sample, the more precise the estimates. But that relationship holds for random samples from the population, not for data generated from a sample. Calculating a confidence interval on a million synthetic records would give a completely false sense of precision.

Something similar happens with data generated by language models. Asking a model to "answer as a 40-year-old customer would" can produce plausible answers, but those answers reflect what the model learned about how people tend to write, not what the real customers of a given organization think. They can help design and test a questionnaire; they do not replace the survey.

The risk of recursion

One phenomenon illustrates these limits particularly well. Recent research showed that when a generative model is trained on data produced by other models, and this is repeated over several generations, quality degrades: the tails of the original distribution disappear and the results become more and more homogeneous. This phenomenon was named model collapse.

From the perspective of this book, the explanation is intuitive. Each generation of synthetic data is a sample from a model that is, in turn, an approximation of the previous data. Extreme values, being infrequent, are less likely to appear in each sample, and what does not appear is not learned. Generation after generation, dispersion shrinks, in the sense studied in Describing variables. The rare, which is often the most interesting part, is lost first.

How to evaluate a synthetic dataset

Before using synthetic data, it pays to evaluate them on at least three dimensions:

  • Fidelity. Do they resemble the real data? The distribution of each variable can be compared with frequency tables and descriptive measures, the relationships between variables can be compared (for example, with correlation coefficients) and, more formally, goodness-of-fit tests like those covered in Nonparametric statistics can be applied.
  • Utility. Do they serve the intended purpose? A common test is to train a model on the synthetic data and evaluate it on real data: if its performance is similar to that of a model trained on real data, the synthetic dataset preserves what matters for that task.
  • Privacy. Do they reveal information about real people? A generative model can memorize and reproduce records from the original dataset, especially the unusual ones. Checking that no synthetic record is identical or nearly identical to a real record is a minimum control.

It is worth noting that these dimensions are often in tension: the more faithful a synthetic dataset is, the more it resembles the real data and the greater the risk that it reveals information about them.

Recap

Synthetic data are a powerful tool for testing, teaching, sharing and exploring scenarios. But they are not observations: they are the reflection of a model, and they cannot contribute information that the model does not have. Used with awareness of that limitation, they expand what can be done; treated as if they were real data, they produce conclusions with an apparent precision they do not have.

Further reading