Buyer's guide / AI PCs

How to run LLMs locally: the 2026 hardware guide

Whether a model runs on your desk comes down to two numbers: how much memory it needs, and how fast that memory is. Here's how to size both -- and the 2026 machines that hit each point on the curve.

By Evan Cole · Last updated July 17, 2026

Running a large language model on your own hardware used to mean a rack of GPUs. In 2026 it can mean a single quiet desktop — but only if you match the model to the machine. Two numbers decide everything: the memory a model needs to load, and the memory bandwidth that sets how fast it generates text. Get the first wrong and the model won't run at all; get the second wrong and it runs, but at a crawl. This guide sizes both, then maps them onto what you can actually buy.

Step 1: how much memory a model needs

An LLM's memory footprint is dominated by its weights, and the weights scale with two things: the parameter count and the quantization (how many bits each weight uses). The rule of thumb:

total memory ≈ (params × bytes-per-weight) + KV cache + ~1–2 GB overhead

Bytes-per-weight is ~2.0 at FP16, ~1.06 at Q8 (8-bit) and ~0.55 at Q4 (4-bit). The quick mental shortcut: at Q4, the GB you need is roughly the model size in billions divided by two. Here's the weights-only footprint — add the KV cache and overhead on top:

Model size Q4 (4-bit) Q8 (8-bit) FP16
7-8B ~4-5 GB ~8.5 GB ~16 GB
13B ~7.5-8 GB ~14 GB ~26 GB
30-34B ~18-20 GB ~34-36 GB ~68 GB
70B ~38-43 GB ~74 GB ~140 GB
100B+ (~120B) ~65-70 GB ~130 GB ~240 GB

Want to skip the math? Use the LLM VRAM calculator — plug in a model size, quantization and context length and it returns the memory needed plus which 2026 machine fits.

Don't forget the KV cache. It holds the context and grows with how much text the model is attending to. For a 70B model at long context it can balloon from ~1 GB at 4K tokens to tens of GB at 128K tokens — at the extreme it rivals the weights themselves. Quantizing the KV cache to 8-bit roughly halves it. Practically: size for weights + a comfortable KV margin, so a 70B at Q4 wants ~48 GB of memory for everyday context, not the bare ~40 GB of weights.

Step 2: capacity vs bandwidth vs compute

Three different limits decide whether — and how well — a model runs. Don't conflate them:

  • Memory capacity decides whether the model fits at all. A 512 GB Mac Studio or a 128 GB GB10 box holds a model that simply won't load on a 32 GB card.
  • Memory bandwidth decides how fast it generates (tokens per second). Text generation is bandwidth-bound — each token streams the active weights through memory once — so tokens/sec tracks GB/s closely. This is the number most spec sheets bury.
  • Raw compute (TOPS / FLOPS) matters most for prompt processing and training, and far less for single-stream generation. A big NPU TOPS number is not what makes a local chatbot feel fast.

The punchline: unified-memory boxes fit enormous models cheaply because they carry a lot of (slower) memory — but at ~256–273 GB/s they generate several times slower than a discrete GPU. A discrete GPU that fits the model runs it far faster (~1–1.8 TB/s) but tops out at 24–32 GB. Apple's M3 Ultra is the middle ground: up to 512 GB and 819 GB/s, at a premium.

Step 3: the 2026 hardware, by category

Hardware Memory Bandwidth Runs (at Q4) Note
RTX 5090 32 GB GDDR7 ~1,792 GB/s* up to ~32B at Q4 (70B only with offload) Fastest per token; capacity-capped
RTX 4090 24 GB GDDR6X ~1,008 GB/s* up to ~30B at Q4 Prior-gen fast pick
RTX 3090 (used) 24 GB GDDR6X 936 GB/s* up to ~30B at Q4 Value 24 GB at high bandwidth
Mac Studio M3 Ultra up to 512 GB unified 819 GB/s 100B+ at Q4 Best capacity + bandwidth combo
Mac Studio M4 Max up to 128 GB unified up to 546 GB/s up to 70B at Q4 Turnkey, fast for its class
Mac mini M4 Pro up to 64 GB unified 273 GB/s up to ~30B at Q4 Cheapest turnkey entry
NVIDIA GB10 (DGX Spark / GX10) 128 GB unified 273 GB/s up to ~200B at Q4 Huge capacity + CUDA stack
Ryzen AI Max+ 395 up to 128 GB unified ~256 GB/s* up to 70B at Q4 Cheapest 128 GB unified

* Bandwidth not stated on the maker's own page — GPU figures are from the TechPowerup database; the Ryzen AI Max+ 395 figure is computed from its 256-bit LPDDR5x-8000 spec and corroborated by independent benchmarks. All memory capacities and interface specs are manufacturer-confirmed. See Sources.

Step 4: which should you buy?

  • You mostly run 7B–32B models and want speed — a discrete GPU wins. A used RTX 3090 (24 GB) is the value play; an RTX 5090 (32 GB) is the fastest single card.
  • You want to run 70B+ models at all, quietly and cheaply — a 128 GB unified-memory box. The Ryzen AI Max+ 395 is the cheapest route to 128 GB; a GB10 machine adds the CUDA stack.
  • You want the biggest models with the least fuss — Apple Silicon. A Mac mini M4 Pro is the turnkey entry; a Mac Studio (M4 Max, or M3 Ultra up to 512 GB) is the capacity-and-bandwidth king.

One machine per family, all verified on Amazon at research time (prices move on these niche systems — re-check the seller and price):

Full picks, verdicts and a memory/bandwidth comparison are in our best AI PC for local LLMs guide, with head-to-head numbers in the AI PC benchmark. For what's shipping vs rumored next, see AI PCs & local-AI supercomputers.

Frequently asked questions

How much VRAM do I need to run a 70B model?

At 4-bit (Q4) quantization a 70B model's weights are roughly 38-43 GB, so you want about 48 GB of VRAM or unified memory to run it with modest context — more for long context, because the KV cache grows with context length. That means a 70B fits comfortably on a 128 GB unified-memory box (GB10, Ryzen AI Max, M4 Max) but not on a 24-32 GB discrete GPU without heavy quantization or offloading to system RAM.

Is a unified-memory box or a discrete GPU better for local LLMs?

It depends on which limit you hit first. A discrete GPU (RTX 5090/4090/3090) has far higher memory bandwidth (~1-1.8 TB/s), so it generates tokens several times faster — but it's capped at 24-32 GB, so it can't hold the biggest models. A unified-memory box (GB10, Ryzen AI Max, big Mac) has much more memory (up to 128-512 GB) so it fits enormous models, but at ~256-819 GB/s it runs them slower. Rule of thumb: pick a discrete GPU if your model fits in 24-32 GB and you want speed; pick unified memory if you need to run 70B+ models at all.

Does the NPU / TOPS number matter for running LLMs?

Much less than you'd think. NPU TOPS mostly help prompt processing (prefill) and small on-device models. Single-stream text generation is bound by memory bandwidth, not raw compute — each token streams the active weights through memory once, so tokens-per-second tracks GB/s far more than TOPS. When comparing local-LLM machines, weigh memory capacity and bandwidth first.

Can quantization let me run a bigger model on less memory?

Yes — that's the main lever. Dropping from FP16 (2 bytes/param) to Q8 (~1) roughly halves the memory footprint, and Q4 (~0.55) roughly quarters it, with a modest quality trade-off that's usually small at Q4 for chat-style use. A 70B model needs ~140 GB at FP16 but only ~40 GB at Q4 — the difference between needing a Mac Studio and fitting on a single 48 GB card.

Sources

Memory capacities, interface widths and TOPS are drawn from manufacturer pages: NVIDIA's DGX Spark (GB10, 128 GB unified, 273 GB/s), AMD Ryzen AI Max+ 395 (128 GB, 256-bit LPDDR5x-8000, 126 total TOPS), Apple's Mac Studio (M4 Max up to 546 GB/s; M3 Ultra 819 GB/s, up to 512 GB) and Mac mini (M4 Pro, 64 GB, 273 GB/s), and NVIDIA's RTX 5090 (32 GB GDDR7). Discrete-GPU bandwidth figures come from the TechPowerup GPU database, and the VRAM rule-of-thumb math from this VRAM explainer; these are not manufacturer figures and are labelled as such above. This page was produced with AI assistance; every confirmed spec was cross-checked against a primary source before publication.

Related buyer's guides

All reviews →

Buyer's guide / AI PCs

LLM VRAM calculator

Enter a model size, quantization and context length to estimate the VRAM or unified memory an LLM needs to run locally — and instantly see which 2026 machines can hold it.

Use the calculator →

Buyer's guide / AI PCs

Mac vs PC for running local LLMs (2026)

It's not Mac vs PC — it's capacity vs speed. A Mac's unified memory fits models a PC GPU can't hold; a PC GPU runs the models that fit far faster. A sourced head-to-head plus which to buy.

Read the comparison →

Buyer's guide / AI PCs

The best AI PC for running LLMs locally in 2026

Six machines for running large language models on your own hardware — from a $799 Mac mini on-ramp to an NVIDIA GB10 supercomputer — ranked by what actually matters: unified-memory capacity and bandwidth, not NPU TOPS.

6 picks →

Buyer's guide / AI PCs

Best CPU for AI and local LLMs (2026)

The honest version: for pure GPU inference the CPU barely matters — so this guide is organised by when it does. Consumer value, the Intel option, unified-memory APUs that hold models a GPU can't, and the workstation platform you only need for multi-GPU rigs.

6 picks →

Benchmark / AI PCs

AI PC benchmark: local-LLM speed and memory (2026)

How the 2026 AI PCs compare for running local LLMs: unified memory, memory bandwidth, NPU TOPS, the biggest model each can hold, and measured tokens per second.

Read the benchmark →

Buyer's guide / PC builds

Best gaming PC under $1,500 in 2026

The honest $1,500 build: a 16GB RX 9060 XT, Ryzen 5 and 32GB DDR5 for strong 1080p-ultra and entry-1440p gaming — with the real shortage-adjusted total, where every dollar goes, and whether a prebuilt beats it this year.

See the build →