How an AI model is trained, from raw text to chatbot

A chatbot is not built in one step. It is built in three, and each one teaches something different. Pretraining gives the model language and a rough map of world knowledge. Supervised fine tuning teaches it to behave like an assistant answering a question rather than a machine continuing a document. Reinforcement learning from human feedback shapes what kind of assistant it is: how it words things, when it refuses, how much it hedges, how eager it is to agree with you. Understanding which stage produced which behavior tells you a lot about what you can and cannot change from your side of the screen.

Stage one: pretraining, where the knowledge comes from

Pretraining is conceptually simple and industrially enormous. The model is shown an unimaginable quantity of text and repeatedly given the same exercise: here is a passage with the next piece hidden, predict it. Each miss nudges billions of internal numbers, called weights, very slightly in a better direction. Repeat that across trillions of tokens.

Nobody labels this data. The text is its own answer key, which is what makes the scale possible. The sources are web pages, books, code repositories, reference works and licensed archives, with heavy filtering to remove duplicates, spam and the worst material. What goes in, and the copyright fights around it, is the subject of where AI training data comes from.

In rough terms, a frontier scale run means thousands of specialized chips running continuously for weeks or months, in a data center built around the power and cooling that requires. The cost is in the tens of millions of dollars and up, which is why very few organizations do it and why most companies build on someone else's pretrained model instead.

What comes out is not a chatbot. It is a text continuation engine. Ask it "What is the capital of France?" and a purely pretrained model might reply with another five quiz questions, because that is what commonly follows a question in the text it read.

Stage two: supervised fine tuning, learning the assistant format

To turn that into an assistant, the model is trained further on a much smaller, much more carefully made dataset: tens of thousands of example conversations, where a human has written what a good answer looks like. Question in, ideal answer out. Same prediction exercise, but now on demonstrations of the behavior you want.

This is where the model learns to answer directly, to structure an explanation, to ask a clarifying question, to say when a request is ambiguous, and to use the chat format at all. It takes days rather than months and a tiny fraction of the compute, but it changes the experience completely.

Stage three: reinforcement from human feedback, learning the taste

Writing an ideal answer for every situation is impossible. Judging which of two answers is better is much easier. That asymmetry is what the third stage exploits.

The model generates several responses to the same prompt. Human raters, following detailed written guidelines, rank them. Those rankings train a second model, the reward model, whose job is to score any answer the way the raters would. The main model is then optimized to produce answers the reward model scores highly.

Why the model is so keen to agree with you

Human raters, like all humans, tend to prefer answers that are confident, well formatted, appropriately flattering and aligned with the questioner's apparent view. Optimize hard against that preference signal and you get a model with a measurable bias toward telling you that you raise an excellent point.

This is not a mysterious emergent trait. It is the direct, predictable output of training on what people said they liked. Labs actively work against it, because a model that caves the moment you push back is less useful, but the pressure is always there. Why AI agrees with you too easily covers how to counter it in your own prompting: ask for the strongest case against your plan, do not reveal which option you already prefer, and treat instant agreement as a signal to check rather than a confirmation.

The same mechanism explains other habits: long answers when short ones would do, bullet lists everywhere, and a certain smoothness that makes wrong answers read as authoritative.

Safety is trained, not hardwired

It is tempting to imagine a list of forbidden topics sitting in a configuration file. That is not how it works. Refusals are trained preferences, learned from examples and rankings, layered on top of a model that has read a great deal of the internet. Most products then add separate filters that inspect inputs and outputs, together with a standing instruction block you never see, quietly placed ahead of your own words every time you open a chat, but the core behavior is statistical.

That has two consequences. Safety behavior is generally consistent but not guaranteed, because it is a learned tendency rather than a rule. And it can be too cautious as easily as too permissive, refusing a legitimate medical question or a piece of fiction, because the training pushed it toward the safer side of an ambiguous line.

What to do with this

When a model disappoints you, name the stage. Missing knowledge means give it the document or use a tool that retrieves one, which is the choice explained in RAG or fine tuning. Wrong tone or too much agreement means change your prompt and ask for disagreement explicitly. And if you are considering training or tuning something yourself, start by checking whether retrieval solves it, because it almost always costs less and updates faster.