Where AI training data comes from and why it matters

A modern AI model is fed from four taps: a bulk copy of the public web, content licensed from publishers and archives, data that users hand over while using the product, and synthetic text written by other models. The proportions vary between companies and none of them publishes a full ingredient list, but the shape is consistent enough to reason about. This matters for one blunt reason: a model has no knowledge from anywhere else. Whatever is missing from the pile is missing from the answer, and whatever is over represented in the pile comes back at you more often than it should.

The four taps and what each one adds

Training a large model consumes text measured in trillions of tokens, a token being the chunk of text models count instead of words. A few trillion tokens is the rough equivalent of tens of millions of books. No licensing budget on earth buys that much edited prose, which is why the open web does the heavy lifting and the other three taps are used to fix what the web is bad at.

What a web crawl actually scoops up

Crawlers work like a search engine's indexer: follow links, fetch pages, save the text, repeat. The largest public archive of this kind has been running for more than a decade and each snapshot covers on the order of a few billion pages. Anyone can download it, which is why it appears in so many training sets.

What arrives is raw. Before anything is trained on it, teams run a pipeline that typically:

1. Strips navigation, ads, cookie banners and boilerplate, keeping the article text.
2. Removes near duplicate pages, since the same wire story can appear on hundreds of sites.
3. Drops pages that fail quality heuristics, such as very short pages, keyword stuffed spam or machine translated filler.
4. Filters out known adult, violent and personal data heavy sources, imperfectly.
5. Mixes in higher quality collections (code, reference works, licensed books) at a deliberately higher weight, so a well written page counts for more than a forum reply.

Two things follow from that pipeline. First, quantity is filtered down hard, often to a small fraction of what was collected. Second, the weighting decisions are editorial choices made by a handful of engineers, and they are one of the least visible reasons two models behave differently.

How the source shows up in the answer on your screen

The pile's shape becomes the model's personality.

English dominates. Roughly half the text in the big open crawls is English, and the highest quality curated collections lean further that way. That is why a model answers a question in fluent English, then gives a thinner, oddly formal answer to the same question in Portuguese or Tagalog, and why it costs more tokens to do so.

Forums are over represented. Public question and answer sites, hobby forums and code repositories are enormous, well structured and free to collect. So models are unusually good at the kind of problem people post publicly (a Python error, a router setting) and unusually confident in the register of a helpful stranger who is sometimes wrong.

Recent and paywalled material is thin. Anything behind a subscription, inside a PDF nobody linked to, or published after the training run ended is simply absent. That absence is the mechanism behind knowledge cutoffs, and it hits local news, court records and trade publications hardest.

Skew becomes bias. If a job title appears mostly beside one gender across millions of pages, the model reproduces that association unless it is trained against it. That is the plumbing behind bias in AI output, and it is a data problem before it is an algorithm problem.

The copyright question, described rather than predicted

Scraping copyrighted work to train a commercial model is being litigated in several countries at once, brought by authors, news publishers, music labels, image libraries and individual artists. The claims usually cover both the copying done to build the dataset and, separately, whether outputs reproduce protected expression.

The legal footing genuinely differs by region. In the US the central question is whether training counts as fair use, which is decided case by case on factors including how transformative the use is and what it does to the market for the original. In the EU there is a text and data mining exception that permits analysis of lawfully accessed content, but rights holders can reserve their rights in a machine readable way, and the EU AI Act separately requires providers of general purpose models to publish a sufficiently detailed summary of training content and to have a copyright policy. Those obligations are covered in what the AI Act asks of general purpose models. The UK and other jurisdictions have been working through their own versions of the same argument.

Practical translation for you: nobody can currently tell you with authority that a given model was trained lawfully, and anyone who states it flatly in either direction is guessing.

When models are fed their own output

Since AI generated text now makes up a growing share of new web pages, the next crawl inevitably contains model output. Train heavily on that and quality degrades in a specific way: rare words, unusual phrasings and minority viewpoints thin out first, the output drifts toward the average, and errors harden into facts. Researchers call this model collapse.

It is avoidable rather than inevitable. Careful use of synthetic data works well when the examples are verified, for instance math problems where the answer can be checked automatically. The danger is undirected scraping of a web that is quietly filling with unverified machine text. This is the commercial reason licensing deals and human written archives keep getting more expensive, and why provenance is now part of how models are trained.

What to check first

Start with your own footprint, then adjust how much you trust the output.

1. Open the data controls in every AI account you use and look for a training toggle. Turn it off if you paste anything you would not post publicly. The steps per service are in how to stop your chats training AI.
2. If you run a website, add rules for the named AI crawlers to your robots.txt, and use the EU style machine readable rights reservation if your audience is there.
3. Before trusting an answer on a niche, local or recent topic, ask yourself whether the source material would plausibly exist in bulk on the open web. If not, verify it.
4. Treat any statistic, quotation or legal citation as unverified until you see it on the original source.

The useful mental model is simple. A model is a compressed average of what it was shown, so ask what it was likely to have been shown before you ask whether it is right.