How a language model actually works: tokens and probability

A language model does one thing over and over: it looks at the text so far and predicts what small chunk of text most plausibly comes next. Then it adds that chunk to the text and predicts again. An entire three paragraph answer about your mortgage is produced that way, one piece at a time, with no plan written down in advance and no moment where the model checks a fact against a source. Once you can picture that loop, most of the odd behavior you have seen from a chatbot stops being mysterious.

Here is the whole pipeline, followed by what each part actually does.

Step one: your sentence is chopped into tokens

The model cannot read letters or words. Before anything else, your text is split into tokens, which are the units the system actually handles: usually a common word, a word fragment, a space plus a word, or a punctuation mark. "The weather in Denver" comes out as roughly four tokens. An unusual surname or a technical term may split into three or four pieces on its own.

Each token has an ID number, and each ID maps to a long list of numbers called an embedding, typically a few thousand of them. That list is the model's representation of what the token means in context: words used in similar ways end up with similar numbers. From this point on there is no text inside the machine, only arithmetic on large grids of numbers. What tokens are and how many fit in a page is worth reading if you are trying to estimate cost or length.

Step two: attention decides which earlier words matter

The signature piece of a modern language model is attention. For every token being processed, the model scores how relevant each earlier token is to it, then blends those earlier representations together in proportion to the score.

Take "The nurse told the patient she would be back in ten minutes." To handle "she" sensibly, the model needs to weight "nurse" heavily and "minutes" barely at all. Attention is the mechanism that does that weighting, and it does it dozens of times in parallel with different learned emphases, some tracking grammar, some tracking topic, some tracking things researchers still cannot label neatly. Stack that operation across many layers, each feeding the next, and the representation of every token gradually absorbs the context around it.

This is why word order changes an answer, why a constraint you gave five paragraphs ago still shows up in the output, and why the model can resolve a pronoun that a simple keyword system would fumble.

Step three: a probability for every possible next token

After the last layer, the model produces a score for every token in its vocabulary, typically 100,000 or more options. Those scores are converted into probabilities that add up to 1. For the prompt "The capital of France is", the distribution might be overwhelmingly concentrated on " Paris", with tiny slivers left for " a", " located" and thousands of others.

The model does not simply take the top option. A sampling step draws from the distribution, and settings like temperature control how adventurous that draw is. Low settings make it stick close to the most likely token, which is what you want for extraction or code. Higher settings let it reach further down the list, which is what you want for brainstorming. This is covered in detail in temperature, top p and the settings that change AI output.

Then the chosen token is appended to the text and the whole process repeats from the top. A 600 word answer is about 800 passes through the model. That is also why answers stream in word by word rather than appearing all at once: each piece genuinely does not exist until the moment you see it.

Why a wrong answer sounds exactly like a right one

Nothing in the loop above is a database query. There is no row containing your local council's phone number that the model can either find or fail to find. Everything, correct or not, is reconstructed from statistical patterns.

That has one consequence worth sitting with: the process that produces a correct citation and the process that produces an invented one are identical. Both are sequences of high probability tokens. The model has no internal signal that says "this part I am sure of, this part I assembled." So the tone of an answer carries no information about its accuracy, which is exactly backwards from how you judge a human expert. See why AI makes things up for what to do about it.

Checking an answer in three steps

1. Ask yourself whether the answer depends on a specific fact, a number, a name, a date, a quote or a legal rule. If it does not, the risk is low.
2. If it does, verify that fact at its source, not by asking the model again. Asking again usually gets the same pattern reproduced with the same confidence.
3. For anything with money, health, legal or safety consequences, treat the model's output as a first draft written by someone who has read widely and remembers imperfectly.

What to check first

Next time an answer surprises you, work backwards through the loop. Was the information ever available to the model, either in training or in what you pasted? Did it fall outside the context window because the chat got long? Was the task really about counting or spelling, where tokens get in the way? Nearly every complaint about chatbot behavior lands on one of those three, and each has a different fix: give it the source, start a fresh chat, or use a tool built for the job.