Distillation: Genius or Theft?

 

n early 2026, OpenAI told a US congressional committee that China's DeepSeek had been "free-riding" on American AI. The technique it named was distillation: training a cheaper model on the outputs of a more powerful one to copy its abilities. Anthropic made a similar charge against several Chinese labs. Much of the Western press reached for the same word. Not innovation. Theft.

Hold that word, because there's an awkward fact sitting right next to it. Those same American labs built their models by scraping the open internet, and with it the copyrighted archives of newspapers, the work of authors and artists, and proprietary text none of them paid for. The New York Times is suing. Hundreds of other publishers are too. OpenAI's defense is "fair use," the legal phrase for we built something new on top of what already existed.

So the principle gets slippery fast. When a Chinese lab learns from an American model, it's theft. When an American lab learns from everyone's writing without asking, it's fair use. Both can't be a clean rule. One of them is just a function of who's holding the lead.

I'll argue something uncomfortable: distillation is not a Chinese trick. It's how innovation has always worked. And whether we call it genius or theft usually depends on which side of the wall we're standing on.

What distillation actually is

Strip away the menace and distillation is closer to apprenticeship than to burglary. The student model never receives the teacher's weights or its training data. It watches the teacher's outputs and learns to generalize from them. If a young engineer studied a master's work, absorbed the patterns, and went on to do similar work more cheaply, we wouldn't call it theft. We'd call it education. When software does the same thing, we suddenly reach for a darker word.

It's also worth noting that many of the Chinese "distilled" models are built on openly released foundations like Llama and Qwen, which were put into the world precisely to be built on. The genuine dispute isn't whether learning-from-outputs happened. It's whose outputs, and under what terms. That's a narrower and more honest question than "theft."

Innovation is a relay race, not a lone genius

We tell a flattering story about invention: the solitary genius, the blank page, the bolt from the blue. It's mostly myth. Newton, no modest man, admitted he saw further only "by standing on the shoulders of giants." Almost everything new is an incremental step on top of someone else's work, often someone in another country, often uncredited.

The irony runs right through AI itself. Every model in this fight, American and Chinese alike, is built on the Transformer, the architecture from a single 2017 Google paper that everyone then copied and extended. The entire industry is one long act of building on a rival's published idea. Three older examples should finish off the myth.

The numbers you do math with were borrowed, then renamed. Place value, the decimal system, and zero as a number were worked out in India; Brahmagupta wrote the rules for zero in the 7th century. Arab scholars absorbed and extended this system. The words "algorithm" and "algebra" both come from al-Khwarizmi and his work. When it reached Europe through Fibonacci in the 13th century, the continent called the digits "Arabic numerals," and India, where they were born, largely fell out of the story. The most basic tool in global commerce is a chain of borrowing in which the original source was written out of its own invention. Nobody now argues Europe should have refused positional notation because it came from elsewhere.

Japan turned copying into a quality empire. For a generation after the war, "Made in Japan" meant cheap imitation. Japanese firms reverse-engineered American cars, cameras, and electronics. Then they took a statistical quality method that US industry had largely ignored, Deming's, and perfected it. The copier became the benchmark the world measured itself against: Toyota, Sony, Canon. Nobody calls Japan's rise theft anymore. We call it excellence. The only thing that changed was the result.

And America climbed the very same way. Britain invented the industrial revolution and guarded it, banning the export of textile machinery and even the emigration of skilled mechanics. So in 1789 a young Briton named Samuel Slater memorized the designs of Arkwright's mills and carried them to America in his head. In Britain he is "Slater the Traitor." In America he is the "Father of the Industrial Revolution." Francis Cabot Lowell did the same with the power loom, touring British factories and rebuilding what he saw from memory. Alexander Hamilton openly urged the young republic to acquire foreign technology by whatever means. American industrial supremacy began as the deliberate copying of a rival who was trying to stop it.

The ladder, and the people who climb it

See the pattern. India seeded the mathematics. The Arab world carried and extended it. Europe took it and built modern science. America copied Europe to industrialize. Japan copied America and beat it on quality. China is now copying America in AI. Each stood on the one before, and each, on reaching the top, was tempted to call the next climber a thief.

Economists have a name for this: kicking away the ladder. You climb using every tool available (copying, borrowing, distilling), and the moment you're on top you develop a deep and sudden respect for intellectual property, then write the rules so the next country can't do what you just did. Britain tried it on America. America is now trying it on China, through export controls and accusations both. The argument always arrives dressed as principle. It is almost always about position.

Where the honest line actually is

I'm not pretending all copying is the same, and the serious version of this argument has to concede the difference. Learning from public work is one thing; deliberately breaking an agreement you signed and using deception to extract outputs at scale is another. If DeepSeek's engineers violated OpenAI's terms of service to do this, that's a legitimate grievance about method, and contracts matter.

But look closely and that is the exact grievance the newspapers have against OpenAI: that it took what it wasn't authorized to take, at scale, and built a competitor on top of it. You don't get to call your own scraping "fair use" and the other side's distillation "theft" from the same set of facts. Either learning-from-the-work-of-others is a legitimate engine of progress, with limits we apply evenly to ourselves and our rivals, or it isn't.

There's a strategic point hiding under the moral one, too. Distillation can shorten the journey, but it can't replace the ecosystem that makes frontier AI: the compute, the chip supply chains, the data pipelines, the talent, the capital. And history is blunt about hoarding: every attempt to lock knowledge in, from Britain's machinery bans to today's chip controls, slowed diffusion a little and spurred the rival's home-grown innovation a lot. The country that wins the next decade won't be the one that litigated hardest. It'll be the one that out-built.

So, genius or theft?

Both, and neither, which is to say the question is the wrong one. It pretends to be about ethics when it's really about power, and about who currently benefits from drawing the line where they've drawn it.

So I'll leave you with this. The next time you hear that a rival "stole" its way to the frontier, ask the older question first: how did the accuser get there? Because almost every great power on that ladder was once the thief in someone else's story.

 

# I Sat In on a Webinar About Teaching AI to Spot Cancer. Here's What a Non-Medical Practitioner Actually Understood

 

Last week I joined a webinar on fine-tuning a vision foundation model to detect cancer in pathology slides. I'm not a pathologist. I can't read a slide, and a good chunk of the biology went over my head. But the machine-learning shape of the problem is something any ML researcher can follow, and by the end I could rebuild the pipeline on paper. This is that explanation, written for people like me who work in AI but not in medicine.

First, what a pathology slide even is

When a doctor removes a piece of tissue, a lab stains it (usually with H&E, which turns cell nuclei purple and other structures pink) and mounts it on glass. A pathologist looks at it under a microscope to judge whether the cells look cancerous.

To bring AI in, the slide gets scanned into a digital file called a whole-slide image, or WSI. The first surprise: these files are enormous. A single scanned slide can run to 100,000 by 100,000 pixels. That is a gigapixel image, hundreds of times bigger than anything a standard vision model takes as input. You cannot feed a whole slide into a network the way you feed it a photo of a cat.

 Figure 1 — A gigapixel slide is cut into thousands of small tiles (patches) before any model sees it.

 

Figure 2 — An H&E-stained whole-slide image. Illustrative; source: # Automated Tumour Detection in Whole Slide Images: An End-to-End Deep Learning Pipeline (https://balintstewart77.github.io/camelyon16-pathology/)

 

The gigapixel problem, and the weak-label twist

The workaround is tiling. You chop the giant slide into thousands of small patches, often 256 by 256 pixels, and treat each patch as an image the model can handle. One slide becomes ten thousand little pictures. Before that, teams run a quick tissue-detection step to throw away the blank glass, and often a stain-normalization step (methods with names like Macenko and Vahadane) so slides from different labs don't look wildly different in color.

Tiling solves the size problem and creates a new one. You now have ten thousand patches per slide, and for most of them, nobody has told you which contain the cancer. The label you actually hold sits at the level of the whole slide: this patient has cancer, this one doesn't. The needle is somewhere in the haystack, and you were handed only the fact that a needle exists. In ML terms this is weak supervision, and it drives the whole design.

What the foundation model brings

Here is where the vision foundation model, or VFM, comes in, and where the webinar clicked for me.

A pathology VFM is a large vision transformer already trained on an immense pile of unlabeled patches. Virchow, one of the well-known ones, is a 632-million-parameter model trained on roughly 1.5 million whole-slide images with self-supervised learning (the DINOv2 approach from general computer vision). UNI, Prov-GigaPath, and CHIEF are other examples. Self-supervised means it learned the visual structure of tissue with nobody labeling cancer versus benign, the same way a language model learns from raw text.

The payoff: this model already knows what tissue looks like. Hand it a patch and it returns a compact numerical fingerprint, an embedding, that captures the meaningful content. You didn't teach it cells, staining, or texture from scratch. Someone spent enormous compute doing that once and released the weights.

Fine-tuning: you train very little

This reframes the task, and it's the part most relevant to non-medical ML people. You are not building a cancer detector from zero. You are adapting a model that already sees tissue clearly.

The webinar laid out three levels of effort:

The lightest and most common approach freezes the foundation model completely. You run every patch through it once, collect the embeddings, and discard the pixels. Then you train a small aggregator that takes the bag of patch embeddings from one slide and produces a single slide-level prediction. Because you only have slide-level labels, this aggregator is a multiple-instance-learning head with attention: it learns which patches deserve attention and downweights the rest. The attention scores come free, and they show you where on the slide the model is looking.

A middle option adds linear probing or small adapters on top, still keeping the backbone mostly frozen.

The heaviest option fine-tunes the foundation model's own weights, usually with a parameter-efficient method like LoRA so you aren't updating all 632 million of them. It costs the most compute and needs the most labeled data. The presenters' honest take: most teams don't need it. The frozen-encoder-plus-attention-head route gets you far, and full fine-tuning mainly pays off when you have a lot of clean, task-specific data.

 

Figure 3 — The common pipeline: the frozen VFM (blue) turns each patch into an embedding; a small attention-based aggregator (green, the only part you train) combines them into one slide-level call.

Where to get data without a hospital

One relief for outsiders: you don't need a hospital to start. Several large pathology datasets are public and openly licensed. Camelyon16/17 covers breast-cancer lymph-node slides, PANDA covers prostate, and TCGA spans many cancer types. They come with slide-level labels, which is exactly what the weak-supervision pipeline expects. OpenSlide is the standard library for reading these gigantic files.

The part they spent the most time on: not the model, the validation

This surprised me, and it's the most transferable lesson. The presenters spent less time on architecture than on how you check the result, because this is where pathology models quietly fail.

The headline numbers look great. Virchow reported an AUC around 0.949 for detecting cancer across seventeen tissue types, and held up on rarer ones. AUC measures how well the model separates positive from negative cases, where 1.0 is perfect and 0.5 is a coin flip, so 0.949 ranks a cancer slide above a healthy one almost every time.

Then came the warnings. Split your data by patient and by hospital, never by random tiles, or patches from the same slide leak between train and test and your score becomes fiction. Validate on slides from a hospital the model never trained on, because a different scanner and a different lab's staining can look foreign enough to fool it. Report more than one AUC: sensitivity, specificity, and especially the false-negative rate at a high-sensitivity operating point, because in cancer, missing a positive is far worse than flagging an extra slide for review. And don't over-trust the pretty attention heatmap; a pathologist has to confirm the highlighted regions are biologically sensible, because a model can land on an artifact or a smudge and still score well.

What I took away as a non-medical practitioner

Strip out the biology and the pipeline is familiar. A giant image gets tiled into patches. A pretrained foundation model turns each patch into an embedding. A small model learns from weak, slide-level labels to combine those embeddings into a diagnosis, and its attention doubles as an explanation a doctor can inspect. The hard part isn't the model; it's proving the model works on slides it has never seen.

The recipe travels well past cancer. Any domain with enormous images and scarce, coarse labels (satellite imagery, industrial inspection, materials science) can borrow it directly. The foundation model does the seeing. You do the smaller, more careful work of teaching it what to decide, and the even more careful work of checking that it decided for the right reasons.

One line the webinar kept returning to, which I'll pass on: none of this replaces the pathologist. The realistic goal is a second reader that flags suspicious slides and points to where it's worried, so a human spends attention where it counts. If you build in this space, that framing matters as much as the AUC.

I walked in unable to read a slide, and I still can't. But I walked out able to build the pipeline and, more usefully, able to tell a good result from a fragile one.

LLMs aren't Uber; they're MoviePass

 

Every investor wants AI to be Uber. Burn cash now, win the market, raise prices later, print money. The losses are an investment in dominance.

It's the wrong story. The honest one is MoviePass and the difference is the whole ballgame if you're betting your costs on cheap AI staying cheap.

Uber's losses had an exit. MoviePass's didn't.

Uber bled billions on purpose. But notice why the bleeding stopped. Once it owned a city - enough drivers, enough riders, a habit - serving one more ride barely cost anything while its pricing power climbed. The subsidy bought a position that got cheaper to hold. Network effects did the work. There was a door marked "profit" at the end.

MoviePass sold unlimited movies for $9.95 a month. The flaw was brutal: every time you used it, they paid the theater near full price for your seat. The more you loved it, the more they lost on you. Scale never fixed the math - it multiplied the loss. Your best customers were your worst customers. It grew itself to death in about a year.

One subsidy had an exit. The other had a cliff.

The line that separates them: for Uber, usage trended toward revenue. For MoviePass, usage was cost.

So which one is an LLM?

Sit with that line, because LLMs fall on the wrong side of it.

Every prompt burns real compute, every single time. Your heaviest users - the ones running agents, generating all day, building their whole workflow on it - cost the provider the most. Using it more makes the math worse, not better. That's the MoviePass shape exactly: usage is cost, not revenue, and the people who love it most bleed you fastest.

Uber's cost per ride fell toward zero as it scaled. An LLM's cost per query is inference, and inference doesn't drop because you signed up more customers. It climbs with how hard each one leans on you. Models do get cheaper per token over time - but agents now spend tokens fifty at a time, and the frontier keeps moving to bigger, costlier models everyone then expects as standard. The cost floor keeps walking forward.

There's no geographic density to defend, either. Weights leak. Open models catch up. The "unlimited intelligence for $20 a month" plan is running the MoviePass playbook with better branding.

You can already watch the same death spiral

This is the part that should make the analogy land. MoviePass didn't die in one step. It died in a sequence - and AI is visibly walking the same one.

MoviePass tried to hold the unlimited promise, then started carving it up: blackout dates on the movies you actually wanted, peak-hour restrictions, "fair use" caps, verification hoops, and finally a hard quota that replaced "unlimited" entirely. Each step quietly admitted the model didn't work.

Now look at AI in 2026. The best models drifting out of the flat subscription and behind higher tiers or API metering. Usage windows and rolling caps. Slower models on the cheap plan, the good one upstream. Limits that tighten right as you grow dependent. None of that is failure of the technology; it's the same forced retreat from "unlimited," just run earlier and more gracefully than MoviePass managed. We're several steps into a seven-step story whose last step, for MoviePass, was the lights going out.

Where the analogy breaks (and it does)

I'd be doing the same lazy thing I'm criticizing if I pretended it's a clean match. Two real differences:

AI has a moat MoviePass never had: switching costs. Once your data, prompts, tools, and your team's habits live inside one provider, you don't wander off for a dollar. That's far stickier than a movie subscription.

And providers have an escape MoviePass fumbled. MoviePass metered late and clumsily, and customers revolted. AI companies are metering early and smoothly, while the product is still loved. Same move, just run better.

So LLMs won't "die" like MoviePass. But the flat price will, the same way unlimited movies did.

What a leader actually does about it

This isn't "AI is a bubble." The technology is real and permanent. The pricing is the temporary part. Three moves:

Budget for the real cost, not the coupon. If your business case only works at today's subscription rate, you don't have a business case. You have a free trial. Model what happens when your most valuable, heaviest use becomes your most expensive.

Don't get hooked on one provider's cheap tier. The more generous it feels now, the more certain it is to tighten. Build so you can swap models without rebuilding the house.

And separate your real moat from the discount you're enjoying. The discount ends. Your data, your workflow, the thing a competitor can't copy - that's what deserves the investment.

MoviePass taught a generation a hard lesson: when the thing you love is sold below what it costs to make, the love is the problem. AI is more durable than that. But the price on your invoice today is a story about someone else's funding round - not about what this actually costs.

Plan for the bill, not the trial.

We're Running Out of Internet to Train AI On. Here's What Comes Next

 

There's a problem the AI labs don't put on the keynote slides. The big language models are running low on fuel.

They learned to write by reading almost everything humans have ever published. Books, code, forums, the whole internet. That well is close to dry. You can't double the size of human writing on demand, and the easy answer - train models on text other models wrote - quietly poisons them. Quality degrades. The industry has a name for it now: model collapse. (I unpacked that failure mode here → Habsburg Jaw in making - Model Collapse in AI.)

So the question that actually decides the next decade isn't "how much bigger can the models get." It's "where does the next training data come from when the internet runs out?"

The most serious answer going right now is this: stop collecting data. Start generating experience. That's what world models do.

A different kind of model

A language model predicts the next word. A world model predicts the next state of an environment, what happens when you turn the wheel, drop the glass, brake on ice. It learns physics and cause-and-effect, then lets a machine rehearse actions inside its own simulation before doing them for real.

The idea is older than the hype. David Ha and Jürgen Schmidhuber published a paper called "World Models" back in 2018, where an agent learned to play a game inside its own dreamed-up version of it. What's new is that the models finally got good enough to matter outside a lab and that the data wall gave everyone an urgent reason to care.

2026 is when it left the lab

Watch what shipped in the last six months, because the timing isn't a coincidence:

🔹 Google DeepMind released Project Genie to the public in January. Type a sentence, walk around a playable 3D world in real time. By May, it connected to Street View.

🔹 Nvidia launched Cosmos 3 on June 1 - an open model built specifically to train robots and self-driving cars inside generated worlds, with a coalition of robotics companies around it.

🔹 Waymo built its own world model in February to create the dangerous driving situations it can't safely film on real roads. Wayve shipped GAIA-3 for the same reason.

🔹 Fei-Fei Li's World Labs opened Marble to the public. Odyssey raised $310M.

And the loudest signal: Yann LeCun bet his next chapter on the claim that scaling language models is a dead end, and that models which learn the structure of the world are the real path forward.

These aren't six companies chasing a demo. There are six answers to the same shortage.

Why this is a business story, not a robotics one

Here's the move that matters for anyone running a company. Real-world data is slow, expensive, and often impossible to gather - a billion driving miles, a million robot grasps, the rare disaster you can't stage. A world model lets you manufacture that experience. Simulate the edge cases by the million, overnight, for the cost of compute instead of years and lives.

If that sounds abstract, connect it to something you already use: the digital twin. A digital twin tells you what your factory or supply chain looks like right now. A world model is the layer that predicts what it does next - so you can test a change a thousand ways before spending a dollar in the real world.

That's why this spreads well past cars and robots: manufacturing lines, warehouse logistics, supply-chain shocks, financial scenarios, anywhere with humanoids or autonomy on the roadmap. The common thread is rehearsal - deciding after you've seen it play out, not before.

The divide it creates

For three years, the AI advantage was about generating content faster. The next advantage is quieter and harder to copy: generating experience. The companies that learn to simulate their own reality will train, test, and decide faster than the ones still waiting to collect data from the real one. That gap compounds the way cloud and data infrastructure did - invisibly, until a competitor is simply moving at a speed you can't match and you can't quite explain why.

LLMs hit a data wall. World models walk around it by building their own.

So the question worth sitting with: when the easy data runs out, will your company still be waiting to collect more - or generating what it needs?

 

Visual Recap 

 

Your AI isn't Stuck on Technology, it's Stuck on You

 

One of the largest banks in the US ran 47 AI pilots last year. I asked a senior exec there how many had changed the real number on the P&L. He thought about it, then said: "One. Maybe."

Forty-seven experiments. One result. That gap is the whole story of AI in 2026, and almost nobody is naming it correctly.

For three years, the questions were about technology. Which model? Build or buy? How do we do RAG? Are we behind? Fair questions in 2023. They've quietly stopped mattering. The models are good enough, cheap enough, and available to everyone, including your competitors. The barrier to building something with AI has fallen to roughly zero.

So why is the promised return still stuck in pilot purgatory?

Because deploying AI is easy, changing how a company works is hard. And the second one is the actual job.

AI doesn't fix your company. It exposes it

The part executives don't want to hear: AI amplifies whatever was already there. Run it on top of a sharp, fast-deciding organization, and it compounds the speed. Run it on top of unclear ownership and slow approvals, and all you've done is generate confusion faster.

The pilot that summarizes contracts works fine in the demo. Then it dies in the org. Legal doesn't trust the output, nobody decides who's accountable when it's wrong, and the old manual process still runs in parallel "just to be safe." None of that is a data-science problem. You can't prompt your way out of an org chart.

That bank's 47 pilots weren't a technology achievement. They were a museum. Lots of impressive exhibits, nothing in production.

Three places the work actually breaks

Decisions move at committee speed while information moves at machine speed. AI now hands a team a forecast in minutes. Then that forecast waits two weeks for a Thursday steering meeting. When insight is instant and the decision is slow, the speed you paid for evaporates inside your own approval chain.

You're measuring effort in a world that no longer rewards it. Hours worked. Headcount. Tickets closed. Number of pilots launched. These tell you a team is busy. If one person now does what five used to, "busy" is the wrong thing to count. Cycle time, decision velocity, cost avoided, a customer kept. Those are the numbers that moved, and most dashboards don't track them.

The work sits between your departments, but your org chart doesn't. The useful AI workflows cut across marketing and analytics, finance and forecasting, product and support. Your structure still has walls exactly where the value wants to flow. So it doesn't flow.

The one comparison worth holding onto

When factories first wired up to electricity, productivity barely moved. The owners had swapped the steam engine for an electric motor and changed nothing else. Same layout, same workflow, same building designed around one central power source.

The gains came decades later, when a new generation of managers redesigned the whole floor around what electricity made possible: machines anywhere, smaller flexible lines, a different shape of work entirely. The technology had been ready for years. The management caught up late.

We are at the "wired up but unchanged" stage with AI. The motor is bolted in. The factory floor hasn't been touched.

What this asks of you

This is the uncomfortable shift. Your job stops being "what can AI do for us" and becomes "what has to change in how we run for any of that to land."

That means deciding, on purpose, which decisions stay human, which become AI-assisted, and which you'll let a system make on its own - and who is accountable for each. It means killing work, not just speeding it up; half your reports and approvals exist only because automation didn't exist. It means promoting the operator who can redesign a process, not only the engineer who can fine-tune a model.

None of that is exciting. It's slower and more political than buying another tool. Which is exactly why it's the real moat. Anyone can buy the model. Few people are willing to take on the management work around it.

The biggest risk to most executives right now isn't getting out-innovated. It's getting out-managed by a competitor running the same models you have - just inside a company built to actually use them.

So the question I'd put to you: where is your AI actually stuck - the technology, or the way your organization decides, measures, and owns the work? Be honest about which one you've been funding.

Habsburg Jaw in making: Model Collapse in AI

 

 AI model collapse is a degenerative process in machine learning where a model's future generations degrade in quality when trained on synthetic (AI-generated) data rather than original human-created data.

On November 1st, 1700, an entire dynasty of kings came to a crashing end with the death of Charles II of Spain. He was physically & mentally disabled and disfigured. A large tongue made his speech difficult to understand; he was bald by the age of 35, and he died senile and wracked by epileptic seizures. He had two wives, but being impotent, he had no children and thus, no heirs. What else do you expect after 16 generations of inbreeding!

How is it happening

To understand model collapse, you have to understand how AI models process probability.

When a generative AI (like an LLM or an image generator) creates content, it calculates the most statistically probable next word or pixel based on its training. Because it favors the "most likely" answer, its outputs naturally gravitate toward the average or the median of what it has learned.

Generation 0 Training Data

Training Data set originates from human-written books, articles, websites, images, audio, video, code, and human conversations.

The model learns the full spectrum of human expression, including rare, weird, and highly diverse "edge cases" (the tails of the statistical distribution).

Generation 1 Training Data

Training Data set mostly originates from human-generated sources and a few incidents of machine-generated data. The machine-generated data is entering the training data set knowingly as well as unknowingly.

Still good, but begins inheriting biases and omissions.

Generation 2+ Training Data

Most of the training datasets originate from machine generation.  The proportion of human-generated data reduces significantly with each generation.

Training data sets become generic, and different models feed each other. This is resulting in the disappearance of rare information, and mistakes get reinforced.

Every real-world human dataset has a distribution - a spread of outputs across many possible styles, topics, phrasings, edge cases, and rare examples. When a model trains on this and generates new content, it approximates that distribution but doesn't reproduce it perfectly. The tails - the unusual, the rare, the surprising - get underweighted. When the next model trains on those outputs, it further smooths out the tails. Over enough generations, you converge on an overly smooth, narrowly peaked distribution. The model becomes increasingly "average" and loses the ability to represent rare-but-real things.

Types of Model Collapse

Distribution Collapse – Loss of Tail Distribution

The model forgets rare but important patterns.

Example: Original data contains 1,000 bird species, but AI-generated data mostly discusses common birds.  This will result in future models forgetting uncommon species.

Error Amplification

Small errors become accepted facts.

Example: Model A hallucinates a historical date, so AI-generated articles repeat it.  Model B is consuming data from Model A’s output. Model B learns it as truth.

Diversity Collapse

As each model will be trained on other models’ output, outputs across Models become increasingly similar. This will result in the same writing style, explanations, examples, and reduced creativity.

Capability Collapse

The model loses nuanced reasoning abilities.

Examples:

• Less robust coding
• Poor edge-case handling
• Weak scientific reasoning

Symptoms of Model Collapse

The symptoms include:

• Loss of Variance: The model output becomes incredibly repetitive. In image generation, for example, all faces might start to look like the same generic, heavily averaged face.
• Disappearance of the "Tails": The model forgets about rare concepts, subcultures, or complex vocabulary.
• Compounding Hallucinations: If Generation 1 makes a slight factual error, Generation 2 treats that error as a fact and amplifies it. By Generation 5, the model is completely detached from reality.
• Perceptual Blindness: The model loses the ability to understand the original distribution of the data, making it impossible for it to "recover" even if human data is reintroduced later.

Is Synthetic Data Always Bad?

The simple answer is No, but….

High-quality synthetic data can be extremely valuable in rare edge cases, safety training, mathematical proofs, coding exercises, and data augmentation

The danger arises when synthetic data dominates the training data set, no grounding in real-world human data, and quality controls are weak.

How the AI Industry is Fighting Back

Because high-quality, original human data is a finite resource (often referred to as "peak data"), the industry is actively developing defenses against model collapse:

• Data Provenance and Watermarking: Developers are working on invisible watermarks for AI text, code, and images. This allows future web-scrapers to automatically filter out synthetic data and only train on verified human data.
• Strict Data Curation: AI labs are moving away from "scrape everything" approaches. They are employing massive teams of humans to curate small, incredibly high-quality datasets of original human work.
• Synthetic Data Filtering: While training exclusively on synthetic data causes collapse, researchers have found that training on a carefully curated mix of human data and high-quality synthetic data can actually be beneficial. The key is ensuring the synthetic data is strictly verified for accuracy.
• Alternative Modalities: Since text and image data are easily polluted, some researchers are looking toward training models on physical world data (like video, robotics, and sensor data), which is much harder for an AI to fake.
• Architectural or training innovations: Techniques that better preserve tails or use reinforcement to avoid plateaus.

Summary

Model collapse is the ultimate bottleneck for AI scaling. It proves that AI cannot replace the need for human creativity and original data. To continue advancing, AI models will always rely on a steady diet of genuine, messy, diverse human input. If the internet becomes an echo chamber of AI talking to AI, the technology will stagnate and degrade.