AI How language models work, from tokens to reasoning
Temperature, top-p and the settings that change AI answers
What temperature, top-p, max tokens and system settings actually do to an AI answer, when changing them helps, and why a lower setting does not mean more truthful.
The short answer
- These settings control only how the next token is chosen from a ranked list the model already produced, so they cannot promote an answer the model scored low.
- Temperature stretches or squashes the gaps between candidates: low makes the leading option nearly certain, high gives lower ranked options a real chance.
- Top-k keeps a fixed number of candidates while top-p keeps as many as it takes to reach a share of the probability, which is why top-p adapts better to confident and uncertain moments.
- Temperature zero produces the most typical continuation, not the most accurate one, and a model will invent a citation at zero exactly as it does at any other setting.
- Max tokens truncates rather than shortens, and seeds make output repeatable only on a best effort basis, so record your settings instead of assuming you can regenerate a result.
Temperature, top-p, top-k and the rest do not change what a model knows, how hard it thinks or how careful it is. They change one thing: how the next chunk of text gets chosen from a ranked list the model has already produced. The model scores every possible next token, the scores become probabilities, and these settings decide whether you always take the front runner or sometimes take something further down. That is the whole job. It explains why turning temperature to zero makes answers repeatable rather than correct, and why no setting on this list will stop a model inventing a source.
Where these settings sit in the loop
A language model writes one token at a time, and each pass through the model ends with a score for every token in its vocabulary, tens of thousands of them. Those raw scores get converted into probabilities that add up to one. A separate piece of code, the sampler, then picks one token from that distribution, and the whole loop runs again with the new token appended. The token by token loop is worth reading once if that sequence is new, and what a token actually is explains why the unit is not a word.
Everything here acts on that single pick. Temperature reshapes the distribution before the choice. Top-p and top-k delete candidates from the bottom of the list. Penalties tilt the scores of tokens already used. Max tokens and stop sequences do not affect the choice at all; they decide when the loop stops.
The important consequence: the ranking comes from the model, and none of these settings can promote an option the model scored low. If the correct answer was never in the running, no sampling setting will find it.
What temperature actually does
Temperature divides the scores before they become probabilities. Dividing by a number smaller than one exaggerates the gaps between candidates, making the leader more dominant. Dividing by a number larger than one squashes the gaps, giving the also-rans a real chance.
Take an illustration. Suppose the model is continuing "the soup was" and its four leading candidates land at roughly 40, 25, 20 and 15 percent. At a low temperature those spread apart, the 40 percent option climbs towards near certainty, and you see the same word almost every time. At a high temperature they converge and the fourth option appears often enough to notice. Higher still and the long tail of barely plausible tokens gets picked, which is where output stops making sense.
Temperature zero is a special case called greedy decoding: always take the highest scoring token, no sampling at all. Providers expose different ranges, commonly zero to one or zero to two, so the same number means different things across products and is worth re-testing when you switch.
Two practical notes. The effect compounds over a long answer, because each unusual token shifts the context for every token after it, so a small increase shows up far more at paragraph length than at sentence length. And high temperature does not make a model creative in any useful sense. It makes it less predictable, which is a poor substitute for an interesting prompt: what separates a good prompt from a vague one does more for quality than any slider.
Top-p and top-k: trimming the list instead
These two cut candidates rather than reshaping probabilities.
Top-k keeps the k highest scoring tokens, discards the rest, then samples among the survivors. Set k to 40 and only the top 40 candidates can ever appear. The weakness is that k is fixed while the situation is not. After "the capital of France is" there is one sensible continuation and 40 candidates is 39 too many. Mid sentence in a description there may be hundreds of good options, and 40 is too few.
Top-p, also called nucleus sampling, cuts on cumulative probability instead. Set it to 0.9 and the sampler adds candidates from the top until their probabilities sum to 90 percent, then samples from that set. Where the model is confident the set might hold two tokens; where it is genuinely uncertain, two hundred. The size adapts to the moment, which is why top-p is the more common control.
Both can be combined with temperature, and that is where people get into trouble, because the interaction is hard to predict. The usual advice holds: change one, leave the other at its default, and test.
The settings that are not about randomness
Max tokens is a hard cap on output length, not an instruction to be brief. The model writes as it always would and generation is cut off at the cap, usually mid sentence. Ask for a short answer in the prompt, and set max tokens as a safety net well above what you expect.
Stop sequences halt generation the moment a specified string appears, which is genuinely useful for structured output: stop at a closing tag and nothing trails the block you wanted. Getting the exact format you asked for covers the prompt side of that problem.
Frequency and presence penalties reduce the score of tokens that already appeared, either in proportion to how often or as a flat penalty for appearing at all. They are a blunt fix for a model looping on a phrase. Set too high, they push it away from words it genuinely needs, which is how a technical answer ends up using three names for the same thing.
Seeds and determinism are the most misunderstood of the group. Some providers accept a seed so that the same prompt and settings tend to give the same output, best effort rather than guaranteed. Results drift because work is batched with other requests on shared hardware, because floating point arithmetic on parallel chips does not always add up in the same order, and because the model behind a product name gets updated. If you need output you can reproduce months later, pin the model version, record the settings with the prompt, and store the result rather than assuming you can regenerate it.
Reasoning modes often restrict or ignore these controls, because sampling behavior is part of how the mode was tuned. What thinking before answering actually does explains what those models change instead.
Why low temperature does not mean true
This is the single most common misreading of these settings. Temperature zero picks the most probable continuation. Most probable means most typical given everything the model absorbed in training. It does not mean verified, sourced or true, and nothing in the sampler is checking.
A model asked for a citation it does not have invents one at temperature zero exactly as it does at temperature one. The invention is in the ranking, not in the pick. Why models make things up covers the mechanism: a plausible shaped answer is always available and never flagged.
There is a subtler cost to running at zero. Variability is information. Ask the same question three times at a middling temperature, get three different names, and you have learned the model is guessing. At zero you get the same confident guess three times, which reads as consistency and is really one roll of the dice repeated. Determinism helps when testing a pipeline and misleads when judging whether an answer is solid.
Settings by task
Most products default to the middle of the range, which is reasonable for general chat. These are starting points, not rules.
| Task | Temperature | Top-p | Why |
|---|---|---|---|
| Extracting fields into a fixed structure | Lowest available | Leave at default | One stray token breaks the parse, and there is no upside to variety |
| Code and configuration | Low | Leave at default | Syntax has one correct shape, and near misses cost you debugging time |
| Summarizing a document you supplied | Low | Leave at default | You want faithful compression, not an interesting reading of the text |
| Routine drafting, such as an email or a product description | Low to middling | Leave at default | Enough variation to sound natural, not enough to wander off the brief |
| Brainstorming names, angles or taglines | High | Trim slightly, around 0.9 | The useful ideas are down the list, and the trim keeps out the nonsense at the tail |
| Fiction, dialogue and varied phrasing | High | Leave at default | Repetition is the main failure here, and accuracy is not the measure |
| Anything you need to rerun and compare | Lowest, plus a seed if offered | Leave at default | Repeatable enough to diff, though never guaranteed identical |
What to change first
In a consumer chat app most of these controls are not exposed, and that is less of a loss than it sounds. The prompt is the stronger lever: being specific about audience, format and length changes output more than any temperature move, and the mistakes that quietly ruin an answer deserve more of your attention. For a quick review before you send, the prompt checker flags the usual gaps.
If you do have the controls, the routine is short. Start from the default. Move temperature only, in steps you can feel rather than a decimal at a time. Run the same prompt three times at each setting, because one sample tells you nothing about a random process. Record the setting with the prompt when something works, since a prompt without its settings is half a recipe. And when an answer is wrong rather than badly phrased, stop turning dials: that is a prompt, retrieval or verification problem.
Common questions
What temperature should I use?
Start with the default and only move it when output has a specific problem. If answers wander, invent detail or vary too much between runs, lower it. If everything comes back flat, repetitive or obviously formulaic, raise it. For anything you will parse or paste into a system, keep it as low as the product allows, since predictability is worth more there than variety.
Does lowering temperature reduce hallucinations?
Slightly and unreliably. A low setting favors the most common continuation, which on well covered topics is more often the correct one. When the model has no real information, the invented answer is the most probable continuation, so a low temperature locks it in rather than removing it. Verification is the fix, not a slider.
Should I change temperature or top-p?
Change temperature and leave top-p at its default. The two overlap heavily, and moving both makes the result hard to reason about or reproduce. Top-p is most useful when you want high variety with a guard against the genuinely bad options at the bottom of the list, which mainly comes up in creative generation.
Why do I get a different answer to the same question every time?
Because sampling is random by design at any setting above zero, so identical inputs produce different picks. Even at zero, output can vary between runs: requests are processed in batches on parallel hardware where arithmetic does not always combine in the same order, and the model behind a product name is updated over time.
Where do I find these settings?
In developer consoles, API calls and some power user interfaces. Consumer chat apps generally hide them and choose a middling default, sometimes offering a named mode such as precise or creative that adjusts them behind the scenes. If you cannot see the controls, put your requirements in the prompt instead, which has more effect anyway.