Running AI on your own device: what you gain and give up

You can run a genuinely useful AI model on an ordinary laptop, provided you are honest about the trade. Memory decides what runs at all, memory bandwidth decides how fast it feels, and quantization is what makes a large model fit on small hardware at a modest cost in quality. What you gain is real: nothing you type leaves the machine, there is no per token charge and no usage cap, and it works on a plane. What you give up is the top of the quality range, especially on long multi step reasoning, plus every part of a hosted product that is not the model itself. For confidential work and high volume routine work that trade is often worth making. For casual questions it is not.

What running a model locally actually means

You download a file of weights, which is the trained model itself, typically a few gigabytes. A runtime program loads those weights into memory and does the arithmetic on your processor or graphics chip. Your prompt goes into that memory and nowhere else. There is no account, no request over the network, no rate limit and no bill. The models you can do this with are the ones published for download, and what open weight really means is worth reading first, because the licenses vary more than people expect.

The common misunderstanding is thinking of this as a private copy of a hosted assistant. It is not. A hosted product is a model plus web search, document parsing, code execution, memory across chats, voice and image generation. Locally you get the model. Everything else is a separate piece you install, configure and maintain yourself.

The four honest reasons to run it yourself

Confidentiality. The strongest reason by far. Client files, medical notes, unpublished work and anything covered by a contract you signed. With a local model there is no retention policy to read and no processing region to worry about, which is a different situation entirely from what happens to a prompt you send to a hosted service.

Volume. Hardware is a one time cost and the marginal cost per request is electricity. If you are classifying ten thousand support tickets, tagging a photo library or extracting fields from a stack of invoices, local processing can be dramatically cheaper than metered billing, and the work is usually easy enough that a small model handles it.

Offline. Planes, trains, field sites, rural connections, and any environment where the network is untrusted rather than merely absent.

Stability. A model you downloaded does not change under you. Hosted models get updated, retired and re-tuned, and a prompt that worked last month can behave differently this month. If you need the same input to produce comparable output a year from now, local is the only way to guarantee it.

Notice what is missing from that list: better answers. For general questions, a free hosted tier will usually beat what fits on your laptop.

Memory decides what runs, bandwidth decides how fast

The weights have to fit in memory, and the arithmetic is simple. Multiply the number of parameters by the bytes used to store each one. At full 16 bit precision that is 2 bytes per parameter, so an 8 billion parameter model needs about 16 gigabytes. Compressed to roughly 4 bits it needs about 5. On top of that, the conversation itself occupies memory that grows as the context gets longer, which can add gigabytes on a long document. Leave headroom for the operating system and your browser, and work from free memory rather than the number on the box. If you are sizing a machine for this, how much RAM you actually need is the place to start.

Speed is a different constraint. Generating each token requires reading the active weights out of memory, so the ceiling is memory bandwidth divided by the size of the model in memory. A 5 gigabyte model on a machine with around 100 gigabytes per second of bandwidth gives a theoretical ceiling near 20 tokens per second, and real output is typically half to two thirds of that. Ordinary laptop system memory sits in roughly that range. Unified memory on the larger Apple chips is several times higher. Dedicated graphics memory is higher again, which is why a GPU with enough video memory feels so much quicker, and why a model that overflows video memory into system memory slows to a crawl.

Two reference points. Comfortable reading speed is around 10 tokens per second, so anything above that feels fine for chat and anything below 5 feels broken. And feeding a long prompt is a separate, compute bound step, so a fifty page document can sit for a while before the first word appears, even on hardware that then generates quickly.

Quantization in plain terms

A model is billions of numbers. Quantization stores each of them with fewer bits: 16 down to 8, 6, 5, 4 or lower. It is the same idea as saving a photograph as a compressed image. The file gets much smaller, most of what you cared about survives, and some fine detail does not.

What that means in practice. Eight bit is close to indistinguishable from the original for most work. Around four bits is the usual sweet spot, with a small measurable loss that shows up mainly on long chains of reasoning, code and obscure factual recall rather than on ordinary drafting. Below four bits, degradation accelerates quickly, and below three it is almost always better to run a smaller model at four bits than a larger one squeezed that hard.

This is also where parameter count stops being a simple ranking. A well trained recent model of moderate size can outperform a much larger older one, for reasons covered in what model size does and does not tell you.

What to expect at each size

Sizes below assume roughly four bit quantization, which is how most people run these.

The honest summary is that a good local model on a normal laptop feels like a capable hosted assistant from a year or two earlier. The gap is narrow on summarizing, rewriting and extraction, and it is widest on problems that need sustained, careful reasoning, which is exactly where models that think before answering pulled ahead. Local versions of that approach exist, but they generate a great deal of hidden working, and on slow hardware that turns a thirty second answer into several minutes.

The maintenance burden nobody mentions

• You are the integrator. Reading your PDFs, searching your files, browsing the web and running code are all separate components you choose and connect. Each one is a small project.
• Model churn. New weights appear constantly, and the pull to re download five to forty gigabytes every few weeks is real. Fix a model, use it for a quarter, then reassess.
• Disk. Model files accumulate faster than anyone expects. Three sizes from two families is already most of a small drive.
• Heat and battery. Generation pins the machine. On a laptop that means fans, a warm chassis and battery drain measured in tens of minutes rather than hours.
• Shorter usable context. A local model's headline context length is often unreachable on your hardware, because holding it costs memory you do not have. Expect to work in smaller chunks, and see why a model loses the start of a long conversation for what happens when you exceed it.
• No safety net. Local models invent things at least as readily as hosted ones, with fewer guardrails and no search to check against, so the usual verification habits matter more rather than less.

The setup worth trying this weekend

1. Check free memory first, not total. Close everything, look at what is actually available, and subtract 4 GB for the system before you go shopping for a model.
2. Start with a 7B to 9B model at four bit quantization. It fits almost anywhere and answers the only question that matters, which is whether local is worth your time at all.
3. Run your own real tasks through it, side by side with whatever hosted tool you already use. The method in comparing assistants on your own work applies here unchanged.
4. Measure two numbers: tokens per second while generating, and seconds before the first word appears on your longest realistic prompt. Under about 5 tokens per second, you will stop using it.
5. If it is too slow, drop a size before you drop the quantization. A 7B model at four bits beats a 14B model at two.
6. Decide the split and write it down. Local for anything confidential and anything high volume and repetitive. Hosted for hard reasoning, current information and anything where you need the surrounding tools.

That written rule is the part that actually protects you. Confidential material usually reaches a hosted service through a moment of convenience rather than a considered decision, and a line you wrote down in advance is what stops the paste.