I Ran a 26-Billion-Parameter Model on a 13-Year-Old Server CPU. No GPU Required.

“Want to run AI? Buy a graphics card first.” Mention local LLMs to almost anyone right now and that’s the answer you’ll get, nine times out of ten. But is it actually true? A small, quiet group of people is doing something more interesting: running a 26-billion-parameter model on a server CPU that shipped in 2012. Let me tell you what that experiment really proves.

A quick note on honesty before we start. This isn’t a topic that’s blowing up on Hacker News or X this month — chatter has been thin. So think of it less as a raging debate and more as a slow, steady bit of tinkering happening in the background. The direction it points, though, is worth pausing on.

Where the “GPU is mandatory” myth came from

The AI-equals-GPU formula was born in training. Building a model from scratch means running thousands of matrix multiplications in parallel, and for that kind of workload GPUs crush CPUs — sometimes by hundreds of times. That gap is real, and it isn’t going away.

The mistake was assuming the same math applies to inference. Inference is the easy part: you take a finished model, ask it a question, get an answer. It’s far lighter than training. Yet we swallowed the “AI = GPU” equation whole and applied it everywhere.

Here’s the thing most people miss: for inference, the real bottleneck isn’t compute — it’s memory bandwidth. Every time the model spits out a single token, it has to pull 26 billion numbers out of memory. The arithmetic is trivial. What decides your speed is how fast you can shuttle that mountain of data around. Which is exactly why an ancient CPU, given enough RAM, can still run the thing — slowly, but genuinely.

Why Gemma 4 26B is the sweet spot

Google’s Gemma 4 is an open-weights model. That means you can download the weights and run them on your own hardware, no strings attached. And the 26B variant sits in an oddly perfect spot: not too small, not too big.

A 70B-class model is smart, but it’ll choke most CPUs. A 7B model runs light but leaves you wanting more from its answers. The 26B lands in between — enough intelligence to do serious work, yet small enough to squeeze into ordinary system memory once you quantize it.

Think of quantization as putting the model on a diet. Numbers that were stored in 16 bits get trimmed down to 4. You lose a little precision, but the footprint collapses. A 26B model compressed to 4-bit drops to roughly 16GB. And a server that’s over a decade old? Those routinely came with 32GB or 64GB of RAM. Plenty of room.

How usable is it, really

Let’s set expectations honestly. Most of the YouTube demos running Gemma 4 locally are using recent MacBooks or GPU-equipped PCs. One video racked up over 13,000 views showing off “free Claude Code on a MacBook and PC, no API key required.” Another walked through Gemma 4 on a dual-GPU rig with 32GB of pooled VRAM. In other words, the mainstream is still very much team accelerator.

So what about pure CPU — and an old server CPU at that? The short version: it works, but it’s slow. Brace for a few tokens per second. That can be slower than a human types. For real-time back-and-forth chat, it’s maddening.

But flip the framing and the calculus changes. What if the job is summarizing 100 documents overnight? What if throughput matters and latency doesn’t? Just let the CPU grind away while you sleep. For that kind of batch work, a forgotten old server turns into a reliable, tireless workhorse.

What this experiment is actually saying

The point was never bragging rights over running AI without a GPU. The point is how far the floor of AI access has dropped.

A few years ago, a decent language model meant hardware costing thousands of dollars, minimum. Today you can pull real intelligence out of a retired server gathering dust in a closet, or a used workstation you grabbed for the price of a nice dinner. No monthly cloud API bill. No sending your data off to someone else’s servers.

The trade-offs are honest and clear. If you want speed, polish, and frontier-grade performance, you still have to open your wallet. But for the countless jobs where “it just needs to run” is the whole spec, the GPU has quietly gone from requirement to option.

So here’s the summary: “AI needs an expensive GPU” is half right. True for building models. Not so true for using them. If there’s an old machine sleeping in your house or your office, it’s worth asking a question. What could I get done if I bolted a 26-billion-parameter brain onto that thing?