Synthetic Data Generation

Imagine you are training an AI to write code.

Where does the training text come from?

At first, the obvious answer: the internet. Public code repositories, forums, documentation, textbooks.

But there are two problems with that answer.

Problem 1: it runs out. High-quality human text and code are finite. The good parts have already been scraped.

Problem 2: it gets legally heavy. Licensing, scraping rights, and provenance constraints make raw web volume harder to use every year.

So labs tried something else

It turned out this often beats adding more raw web text.

Three concrete milestones define the shift:

• Teacher-model data filtering (2023-2024): Microsoft's phi-1 line showed that a small, carefully "textbook-quality" synthetic/filtered corpus could let a 1.3B model punch far above its size on HumanEval (a code benchmark). The recipe is "use GPT-4 as a classifier/rewriter over public data" rather than "use more data".
• Llama 3 (July 2024): Meta disclosed using GPT-4-generated synthetic data for alignment and quality uplift, and heavy synthetic code data for pre-training refinement, while noting that distillation-style losses from a closed teacher are a licensing concern.
• Reasoning models (2024-2026): The dominant post-training signal is now model-generated long chain-of-thought (step-by-step reasoning text) filtered by verifiers — programs that check answers: code execution, proof checkers, symbolic math. DeepSeek-R1's public recipe and its distillation releases (R1-Distill into Qwen/Llama sizes) are the archetype: generate many reasoning traces, keep only those that reach a verifiably correct answer, then train on them.

The strategic consequence in one line:

Data quality control moved from scraping heuristics into model-in-the-loop generation plus programmatic verification.

Synthetic data is simply

Think of it as a factory with five stations:

1. The seed. Something to start from: public web text, code, math problems, or even nothing (some generators need no seeds — more below).
2. The teacher. A strong model that rewrites, expands, answers, or invents examples. In one plain sentence: the teacher is the model you ask to produce candidate training data.
3. The verifier. A check the candidate must pass — run the code against tests, check the math answer, compare against the official proof. In plain words: the doorman.
4. The blend. Survivors are mixed with real human data, and the ratio per source is tracked.
5. The student. The model being trained on that blend. Here is the twist: a newer, stronger student can become the next teacher. The factory feeds itself upward.

Notice what the factory is NOT: a pile of everything the model said.

code
Copy
generate 100%  →  verify  →  accept maybe 10-40%  →  dedupe  →  blend  →  train
                    │
                    └── the other 60-90% is thrown away

That rejected majority is not a bug. The accept/reject decisions are the actual product. Generation is a commodity; trustworthy filtering is the scarce skill.

Let's make the loop concrete with small numbers.

You want your student model to get better at a coding task. One prompt:

"Write a function that reverses words in a sentence, keeping spacing."

Step 1 — rollouts. Ask the teacher to generate k = 8 different solutions for the same prompt. (One attempt per prompt is weak; many attempts give the verifier something to choose between.)

Step 2 — verify. Run all 8 through a programmatic oracle: compile, execute against hidden unit tests. Say 3 pass, 5 fail.

• Passes → score 1.0. Failures → score 0. The verifier is a program, not an opinion.

Step 3 — choose among the survivors. All 3 correct traces pass, but they are not equal: one is 40 lines of rambling, one is 25 lines, one is 11 lines. Keep the shortest correct one. Why? Training on the rambling one teaches "overthinking"; the concise correct trace teaches the skill plus good taste.

Step 4 — decide about the prompt itself. If all 8 had failed, skip the prompt entirely. Training on junk is worse than training on less.

Step 5 — log everything. Teacher version, decoding parameters, prompt template, pass rate (3/8 = 0.375), number of rollouts. This metadata is what makes the dataset reproducible and debuggable.

Do this across tens of thousands of prompts, and you have an SFT (supervised fine-tuning) dataset built by the factory instead of by annotators.

The same five steps work for math: replace unit tests with an answer checker or proof checker. Nothing in the recipe is code-specific — the only requirement is a check you can run.

Picture the whole pipeline as a funnel. Width = volume of text; the narrow neck is the verifier.

code
Copy
   ╲                                    ╱     seed corpus + prompts
    ╲  teacher generates candidates     ╱      (cheap — millions/day)
     ╲_________________________________╱
      ╲   verifier stack: tests,     ╱        ← MOST GPU DOLLARS SPENT HERE
       ╲  exec, proofs, judges      ╱
        ╲_________________________╱
         ╲  dedupe + decontaminate ╱          ← strip near-duplicates,
          ╲______________________╱              remove eval-set overlap
           ╲  blend with real   ╱
            ╲  human data ___  ╱
             ╲_______________╱
                  student

Follow the volume through it:

• Input: 100% of generated candidates.
• After verification: usually 10-40% survive (60-90% discarded).
• After dedupe and contamination checks: smaller again.
• What enters training: only the tip.

Now the key question the funnel answers:

Not from the wide top. Two datasets of the same size differ in quality almost entirely because of what the neck let through. A weak neck (a flattering judge, or no check at all) turns the funnel into a sprinkler — it just distributes the teacher's mediocrity at scale.

One more arrow worth drawing: the student at the bottom eventually points back to the top (the dashed line in the diagram). Each generation can become the next teacher — which is exactly why the risk section below matters so much.

Carry one analogy through the rest of this topic: a famous chef writing a recipe book for apprentice cooks.

The old way of training cooks: collect every recipe card in every kitchen in the city (scrape the internet). Many cards are greasy, wrong, or written by someone who cannot cook.

The new way:

1. The chef (teacher model) dictates hundreds of recipe cards a day — rewriting old cards, inventing new ones, adapting dishes for different tastes.
2. Before any card enters the official book, the test kitchen (verifier) cooks it exactly as written. If the dish fails, the card is thrown away. No exceptions, no opinions — the test kitchen is the ground truth.
3. Among cards that all produce a correct dish, the editor keeps the clearest, shortest version.
4. The final book mixes chef-written cards with a floor of trusted family recipes (real human data) — and labels how many of each went in (source ratios).

Two lessons this analogy makes unforgettable:

Everything that follows — the generator taxonomy, the verifier stack, model collapse — is variations of this factory: who writes the card, how the test kitchen checks it, and what happens when the chef starts cooking only from their own book instead of real kitchens (that last one is collapse; we will come back to it).

Practitioner vocabulary maps to a handful of reproducible techniques. Each is just a different way of getting the chef to write cards.

1. Seed-based instruction synthesis (Self-Instruct style). Sample K instructions from a prompt template with a few human exemplars ("here are 3 example tasks, write 100 more"), generate responses, filter by lexical overlap. Cheap, diverse, but biased toward the teacher's style.

2. Complexity/depth evolution (Evol-Instruct / WizardLM style). Iteratively rewrite a prompt to be harder, longer, more constrained, or multi-step. Example rewrite: "reverse a string" → "reverse a string in place, then handle Unicode emoji, then write tests". Grows tail difficulty without new seeds.

3. Alignment-set extraction (Magpie style). Feed the model only the pre-user-content chat template — the scaffolding that comes before any human message — and let it emit user queries itself, then generate responses. No hand-written seeds at all, and the query distribution naturally resembles the teacher's instruction distribution. Clever trick: the template itself is the seed.

4. Reasoning-trace distillation. Prompt a strong model for step-by-step solutions, optionally with "refine", "verify", "solve it three ways" scaffolds; keep traces that are correct and (increasingly) that self-correct (the trace notices its own mistake and fixes it — exactly the behaviour you want to teach). This is the main data source for small reasoning models.

5. Persona- and tool-conditioned synthesis. Generate multi-turn conversations conditioned on a persona, locale, tool set, or adversarial user model ("you are a impatient user who keeps changing requirements"). Essential for agent/agentic SFT data: function calls, long-horizon plans, error recovery.

6. Simulated environments and world models. Physics/simulation and world-model video for embodied and physical AI (e.g., NVIDIA Cosmos-style pipelines), where the "text" is an observation-action trace: what the robot saw, what it did, what happened next.

Two cross-cutting habits:

Accept/reject decisions are what turn cheap generations into training signal. Common verifiers, roughly in order of trustworthiness — best first:

• Programmatic oracles (gold standard): unit tests, compilers, interpreters, formal proof checkers (Lean/Isabelle — programs that mathematically validate a proof), SQL execution, constraint solvers. These underpin RLVR (RL with verifiable rewards) and rejection-sampling SFT. Why gold? A test cannot be sweet-talked.
• Consistency oracles: sample N independent solutions; keep answers that agree (self-consistency), or keep traces whose final answer matches a majority vote. Cheap, no extra model. The bet: 8 attempts agreeing by coincidence is unlikely.
• Reward / preference models: a trained judge scoring responses, or pairwise preference against the current model. Useful for open-ended domains (writing, advice) where no program can check — but this is where reward hacking enters: the model learns to satisfy the judge, not the task. Judges like long, confident, well-formatted answers; the factory then produces long, confident, well-formatted wrongness.
• Human spot audits: small, stratified human review of accepted samples is the only reliable way to detect systematic judge failures before you train on millions of examples. "Stratified" = sample from every topic/difficulty slice, not randomly, so rare failures are not averaged away.

A widely used 2024-2026 pattern is filter-then-rerank-then-truncate: dedupe against eval sets (contamination control — see below), rank by verifier score, and cap per-source volume so no single teacher idiom dominates the dataset.

Here is the reject-sampling loop in code — the workhorse function of the whole factory:

import numpy as np

def build_sft_rows(prompts, teacher, verifier, k=8, min_score=1.0):
    rows = []
    for p in prompts:
        traces = [teacher.generate(p, max_tokens=16_000, temperature=1.0)
                  for _ in range(k)]
        scores = np.array([verifier(p, t) for t in traces])   # 1.0 == passed tests
        kept = [t for t, s in zip(traces, scores) if s >= min_score]
        if not kept:
            continue                       # all failed: skip prompt, do not train on junk
        # keep the shortest correct trace: teaches conciseness, limits overthinking
        best = min(kept, key=lambda t: len(t))
        rows.append({"prompt": p, "completion": best,
                     "source": "teacher-v1", "n_rollouts": k,
                     "pass_rate": float(scores.mean())})
    return rows

The factory has four well-documented poisons. Back to the chef analogy: each one is what happens when the test kitchen gets lazy or the chef stops cooking in real kitchens.

• Model collapse / self-consumption. Train repeatedly on model output and the distribution narrows. Tails — rare idioms, unusual code styles, low-resource languages — erode first, and long-run quality can degrade. Why tails first? Generators under-produce what is statistically rare; retraining on the narrowed output makes rare things even rarer, generation after generation. Mitigations: always retain a real-data floor, mix fresh human data, and monitor diversity statistics rather than only benchmark scores.
• Contamination. Synthetic answers derived from a teacher that saw the benchmarks inflate evals: the student is not learning to solve, it is memorizing the teacher's exposure to the test. Control by removing any sample with high n-gram/embedding overlap with held-out sets, and by maintaining live ("contamination-free") benchmarks with freshly written problems.
• Homogenization of style. One loud teacher produces thousands of students that all answer the same way, reducing ensemble diversity — a real problem for agent systems that rely on disagreement (many independent voters only help if they actually differ).
• Provenance and licensing. Synthetic output derived from a licensed/closed model may be contractually restricted; regulators increasingly expect data lineage. The EU AI Act's GPAI (general-purpose AI) transparency obligations push labs toward documented training-data summaries — so "who taught what to whom" becomes a legal record, not a nice-to-have.
• Silent distribution drift. The verifier is a model too. Judge updates change accept rates, so the training distribution shifts under you unless verifier versions are pinned. Your dataset this month is not the same dataset it looked like last month.

At scale, a production synthetic-data platform stops looking like prompting and starts looking like an ETL + batch-inference system:

• Prompt banks and persona catalogs as versioned datasets (checked into storage like code).
• A batch inference fleet (vLLM/SGLang serving) with prefix-cached system prompts and sampling parameters — "prefix cached" means the long shared prompt is computed once and reused, which matters enormously when the teacher answers 100k prompts that start the same way.
• Verifier workers running untrusted code in sandboxes (gVisor/Firecracker — isolated micro-VMs, network-locked), because "run the generated program" means "execute arbitrary code written by a machine for a living".
• A dedupe/contamination stage (MinHash/embedding clustering).
• A data registry with lineage from sample → teacher checkpoint → prompt template → verifier version. When a benchmark regresses, you bisect through that chain.

Cost levers that matter most: prefix caching of long system prompts and seeds; speculating small drafters for boilerplate rewrites (a cheap model drafts, the expensive one verifies the tokens — see speculative decoding); using a strong model only for hard prompts (routing/cascades — cheap teacher first, escalate only failures); and aggressively capping rollouts once pass-rate plateaus (rollouts 9-16 of a prompt family rarely flip the verdict).

The practical heuristic, worth memorizing: if a generator cannot be paired with a verifier you trust more than the generator, do not scale it. Unverified synthetic volume mostly buys noise and drift. (The verifier can be a program — that is the entire premise of RLVR-era data.)

And the one-line design rule: build the test kitchen before hiring more chefs.