Skip to content
Pricing the fourth bit: every route to 128GB for local AI
~/justmeandlinux

Writing / Topics / AI

Pricing the fourth bit: every route to 128GB for local AI

/ AI / 20 min read / 3,866 words

Ask how much memory you need to run large models locally and you get one answer: 128GB. It is repeated so consistently that it has stopped sounding like a claim and started sounding like a spec. I went looking for where the number came from, and the honest version is more interesting than the slogan — because once you take it apart, 128GB turns out to be a ceiling you mostly cannot reach, and the thing your money actually buys is one extra bit of precision per weight.

This post is the concept map I wish I'd had before pricing any of it: what a mixture-of-experts model does to the capacity question, why the number printed on the box is not the number your GPU can address, why the inference engine matters more than the hardware brand, and where the published numbers disagree with each other.

Up front, so there is no ambiguity: I have not run any of this. I do not own a 128GB box and I did not benchmark one. Everything below is a synthesis of vendor specs, project documentation and community reports from late 2026, cross-checked where I could and flagged where I could not. The numbers are November 2026 and several of them are moving weekly, because system memory pricing is in a spike. Treat the prices as a snapshot and the benchmarks as claims.


First, the five words

Skip to the next section if these are familiar. But the whole post rests on them, and three get used wrongly all the time.

A weight. A model is a very large pile of numbers, learned during training. Each number is a weight. Flash-Next has 125 billion of them. That is genuinely what a model is — everything else is machinery for multiplying them together.

4-bit, 3-bit — "quantization". How much room you give each of those numbers. Straight out of training each weight takes 16 bits. Quantizing rounds them down to fit in fewer:

16-bit   65,536 shades    354 GB
 4-bit       16 shades    ~98-120 GB
 3-bit        8 shades     92 GB
 2-bit        4 shades     81 GB

Think crayons. Same drawing, 16 crayons instead of 65,536 — it still looks like the picture, the shades just get rounder. Fewer bits means a smaller file, less to read per token, and therefore more speed; it also means worse answers somewhere, and where exactly is the question nobody has answered.

⚠️ "4-bit" is an average, not a setting. These formats keep some layers at higher precision than others. That is why IQ4_XS at 97.7 GB and Q4_K_M at 119.6 GB are both called 4-bit and are 22 GB apart.

A token. Roughly a word-piece, about ¾ of a word. Models read and write in tokens, which is why every speed here is in tokens/second and every context length is in tokens. The 262,144-token context window is somewhere around a 600-page book.

Mixture of experts (MoE). A normal "dense" model pushes every word through the entire network. An MoE splits the network into many small sub-networks — experts — and a small router picks a few of them per word. Flash-Next has 512 experts and uses 10 of them plus 1 always-on, per token.

Picture a hospital with 512 doctors. Reception sends each patient to 10 of them. You still have to employ all 512 — they must be in the building — but any one visit only occupies 10. Employing all 512 is capacity; the 10 you actually use is bandwidth. That distinction is the engine of this entire post.

⚠️ An "expert" is not a topic specialist. There is no "French expert" or "maths expert" — they are learned mathematical groupings with no human-readable meaning, and the router chooses fresh for every single token rather than once per question.

The inference engine, or "runner". The model file does nothing on its own; it is a pile of numbers sitting on disk. The engine is the program that loads those numbers and performs the multiplication that turns your prompt into text. The model is sheet music, the engine is the musician — same score, different player, very different performance.

The hard part is not the arithmetic, it is logistics: which weights sit on the GPU, which wait in system RAM, when to move them, what to batch together. Different engines answer that differently, which is the whole reason the same machine can produce both 113 and 1,227 tokens/second.

Engine What it is
llama.cpp the general-purpose one; runs almost everything
Strata specialist in expert offloading, aimed at small GPUs
Gufo a tuned fork for AMD Strix Halo
Halogen another Strix Halo engine, closed source

Gufo, for instance, is simply a group who decided llama.cpp was leaving speed on the table on that one specific chip, and rewrote the parts that were.


Where 128 came from

One model. Qwen3.8-Flash-Next — a 125B-parameter sparse mixture-of-experts model that has become the default answer to "what do I run on a big-memory machine?" on r/LocalLLaMA.

The reason 128 is the magic number is dull and real: the 4-bit builds don't fit in anything smaller. Here is the quantization ladder, with file sizes:

Format File size Comment
IQ1_S (1-bit) 70.1 GB
Q2_K (2-bit) 80.9 GB fits a 96GB pool
Q3_K_M (3-bit) 92.0 GB fits a 96GB pool, just
IQ4_XS (4-bit) 97.7 GB the cheapest fourth bit
Q4_K_M (4-bit) 119.6 GB
Q5_K_M (5-bit) 134.7 GB over 128 before any context
Q8_0 (8-bit) 188.3 GB
BF16 354.0 GB multi-GPU node

So the logic writes itself: get 128GB, load the 4-bit file, done. Three things break that logic, and they are the three concepts worth actually learning.


Concept 1 — in a sparse MoE, capacity and compute are separate purchases

The intuition that's wrong: a 125B-parameter model needs all 125B parameters sitting in fast memory to run fast.

What a sparse MoE actually does: with 10 experts firing out of 512, across 48 layers, the work comes to roughly 6B active parameters out of 125B for each token written. There is also a 51B n-gram embedding table and a 4B prediction head bolted on, so the full stored footprint is around 180B parameters — but the per-token effort is closer to a 6B dense model than a 125B one. A model that is enormous to store and small to run.

Two different resources, and people routinely conflate them:

CAPACITY — how much must be REACHABLE
  ~all 125B of it, because the
  router can pick any expert on
  the very next token.

BANDWIDTH — how much is READ per token
  ~6B params' worth: a couple of
  GB at 3 bits. This is the one
  that sets tokens/second.

A dense 125B model reads all 125B weights for every single token, so capacity and bandwidth are effectively the same number and you genuinely need it all in the fastest memory you have. A sparse MoE breaks that coupling. The full file still has to be somewhere the chip can get to, but only a small slice of it moves per token.

Why this matters commercially: "enough memory to hold the model" and "enough memory bandwidth to run it fast" stopped being one purchase. You can satisfy the first with cheap, slow DDR5 and the second with a modest GPU — if your engine is smart about which parts live where.


Concept 2 — expert offloading

That "if" is a whole category of software. The technique is called expert offloading: keep the active, hot path on the GPU and let the dormant experts live in system RAM (or on NVMe), paging them in when the router calls for them.

Strata is the engine built specifically around this for Flash-Next — MIT licensed, C++/CUDA, reusing parts of llama.cpp and ggml, with builds for NVIDIA, AMD (Strix Halo included), Intel Arc and Mac. The model card itself notes the n-gram lookup table is more amenable to offloading than the rest of the model, so the model design and the engine are meeting each other halfway.

The analogy that made it click for me: a 128GB box is a desk big enough to hold every book you own open at once. A GPU-plus-system-RAM setup is a smaller desk next to a bookshelf. If each word you write only needs a few pages, the shelf is fine. You are paying for desk area you never use.

The reported consequence is the part that reframes the whole buying decision. One community report has Flash-Next running on an RTX 3090 — a 24GB card from 2020 — using 23GB of VRAM plus 47GB of system RAM. Nothing on that machine is 128GB, and the model runs.


Concept 3 — the 128 on the box is a ceiling, not an allocation

This is the one that would have cost me real money. On a unified-memory machine, "128GB" is the total system memory. The amount the GPU can address is a separate, smaller, and often configurable number. Every row below is a 128 GB machine:

Platform GPU can use How to change it
Strix Halo, Windows ~96 GB Adrenalin → Tuning → Variable Graphics Memory
Strix Halo, Linux 100 GB+; ~110 GB seen GTT kernel param + BIOS framebuffer
Apple Silicon ~96 GB (⅔–¾ of RAM) iogpu.wired_limit_mb — see below
NVIDIA DGX Spark ~122 GB —

Now hold that against the quant table. A 96GB addressable pool clears the 3-bit build at 92.0 GB and misses every 4-bit build, the cheapest of which is 97.7 GB before you allocate a single token of context. The native context window here is 262,144 tokens, and KV cache sits on top of the weights.

Three consequences worth internalising:

  • On a Mac, raise the wired limit on day one. The developer of one of the main Mac inference engines tells you to do this before running anything at full context:

    # 120 GB of a 128 GB Mac
    sudo sysctl iogpu.wired_limit_mb=122880
    

    It resets on every reboot; a LaunchDaemon is the documented way to persist it. Leave headroom — around 14GB on a 128GB machine is the usual advice — or you will wire memory the OS needs and the machine will behave very badly.

  • On Strix Halo, Linux is not a preference, it's capacity. The Windows 96GB default is the difference between running 4-bit and not.

  • Even the expensive box offloads. NVIDIA's own 4-bit file for this model is reported at 124GB against ~122GB usable, so the popular recipe for the $6,950 DGX Spark leaves part of the model on the SSD. Microsoft's new machines acknowledge this in a footnote saying GPU-addressable memory is less than total — without saying how much less.


Concept 4 — prefill and decode are two different speeds

Before the prices, one more distinction, because benchmark claims are meaningless without it.

PREFILL  (prompt processing, "pp")
  Reading the prompt you just sent.
  Compute-bound, highly parallel.
  Thousands of tokens/second.
  You wait on this when an agent
  re-reads a 160k-token context.

DECODE  (token generation, "tg")
  Writing the answer, one token at
  a time. Memory-bandwidth-bound.
  Tens of tokens/second.
  This is what you watch scroll.

Most "X tokens per second" headlines quote decode, which is the smaller and less interesting number if your workload is long-context agent work. There, prefill dominates wall-clock.

And prefill is where the engine turns out to matter more than the hardware. On the same Strix Halo machine with the same model at 160,000 tokens of context, one report has the default engine reading the prompt at 113 tok/s and a tuned community fork — Gufo, which bills itself as the open-source Strix Halo AI engine — at 1,227 tok/s. Same box, same weights. 23 minutes versus 2 minutes, entirely down to software. Halogen posts similar prefill figures and is popular with owners, though it's closed source.

If you take one practical thing from this post: pick your engine before you pick your brand. The 4 tok/s that sends people shopping and the 50 tok/s they're shopping for are frequently the same hardware.


The routes, priced

Snapshot of early November 2026. Decode figures are for Flash-Next at the stated quantization, as reported by owners and project docs — not measured by me.

Route Price Decode
Strix Halo mini PC $3,099 47–54 tok/s @ 4-bit
Gaming PC + 64GB RAM ~$880 50–62 tok/s @ 3-bit
4× used Radeon Pro V620 ~$3,000 60–70 tok/s @ 4-bit
Mac Studio M5 Max 128GB ~$5,099 ~40 prose / 75 code
RTX Spark laptop/desktop $5,899+ not published
NVIDIA DGX Spark 64GB $4,999 —
NVIDIA DGX Spark 128GB $6,950 ~41 tok/s median

And the catch on each, which is where the actual decision lives:

  • Strix Halo mini PC — $3,099, 256 GB/s. The cheapest honest route to 4-bit, but only on Linux; the Windows 96GB default puts every 4-bit build out of reach. Note this is one chip in several boxes — Framework wants $3,449 (currently out of stock) and Minisforum $3,799 for the same Ryzen AI Max+ 395 at the same bandwidth. You are paying for the chassis, not the silicon.
  • Gaming PC + 64GB RAM — ~$880. Matches the expensive boxes at 3-bit and beats all of them on prefill. Cannot reach 4-bit, and it's a one-person machine. The whole argument of the next section.
  • 4× used Radeon Pro V620 — ~$3,000. Fastest decode on this list, but 500–600W, and the deal is gone — V620 prices climbed to V100 levels once the build went viral.
  • Mac Studio M5 Max 128GB — ~$5,099, 614 GB/s. More than twice Strix Halo's bandwidth, and the 40-core GPU config is the only path to 128GB. But nobody has published a prefill number for this model on this machine, and that is a genuine hole rather than an oversight on my part: for long-context agent work, prefill is what decides whether the machine is pleasant to use.
  • RTX Spark laptop/desktop — $5,899+. Not the DGX Spark, despite the name and the three-week gap between launches — a different chip entirely. Reported bandwidth of ~273–307 GB/s puts it next to Strix Halo, not next to the Mac Studio, and the vendor's speed claims were measured on pre-production 64GB units.
  • DGX Spark 128GB — $6,950. Up roughly 75% from launch on memory pricing. Its old advantage was prefill, and current engines on cheaper hardware have narrowed that to almost nothing. The 64GB model at $4,999 is not a route to 128GB: two of them is $10,000.

A note on the multi-GPU route, since the power bill is part of the price: 500–600W under load at 4 h/day and $0.15/kWh is roughly $120/year, against about $38/year for a Strix Halo box doing the same work. Over three years that is most of a GPU. The current alternatives don't rescue it either — 4× Intel Arc Pro B70 or AMD Radeon AI Pro R9700 lands near $6,800, 4× used Instinct MI50 is $2,300–2,600 but runs on a chip AMD put in software maintenance mode years ago, and a single RTX Pro 6000 Blackwell with 96GB is $8,900–$13,250 if you can find one.


The route nobody in the 128GB conversation prices

Strata's own documentation lists a test setup that is deliberately unremarkable: a 12GB consumer GPU, a six-core Ryzen, 64GB of system RAM. On the 3-bit build it reportedly writes 50–62 tok/s and reads a 32,000-token prompt at ~1,750 tok/s.

That is the decode speed of the $3,000–$7,000 boxes with faster prefill than any of them, on a card a lot of people already own.

So here is the actual price of the fourth bit:

ALREADY OWN A 12GB CARD?
  64GB of DDR5         ~$880    3-bit
  Strix Halo box       $3,099   4-bit
  ───────────────────────────────────
  The fourth bit     ~$2,200

BUILDING FROM SCRATCH?
  GPU + RAM + a PC    ~$2,400   3-bit
  Strix Halo box       $3,099   4-bit
  ───────────────────────────────────
  The fourth bit       ~$700

$700 is a number you can reason about. $3,000 for "the proper way to do local AI" is not.

Throwing more RAM at the problem does not close the gap, either: one owner with a 5090 laptop GPU and 192GB of system RAM reports 53–59 tok/s at 3.5-bit but only 21–33 tok/s at 4-bit — roughly half. The PC route matches the big memory pool at three bits and falls behind at four. Every option in this post lands on one side of that line.


Where the published numbers disagree

This is the part I'd want someone to tell me, and it's why I'm not giving you a confident recommendation dressed up as a verdict.

The benchmark figures are not consistent across sources. Community reports put a 12GB card at 50–62 tok/s on the 3-bit build. Strata's own comparison page, on a slightly older 12GB card, shows 21.4 tok/s. Different card generation, different quant, different context depth, possibly different engine build — and none of the sources pin all four down. A 2× spread between two reports of "the same thing" means the thing isn't the same.

Nobody has published the quality comparison that matters. The lab behind the 3-bit build claims it retains 99.4% of the full model's score across three benchmarks — but on at least one of those the full model already scores 100%, so "99.4% of perfect" is doing less work than it sounds like. Owner opinion splits hard: some call the 2- and 3-bit builds lobotomised, others run 4-bit across three 3090s and swear by it. No one has run both quantizations side by side through a long agent session, which is the only workload that would justify buying 128GB in the first place. That gap is the single biggest unknown in this whole decision.

On a strict reading of the quant table, nothing fully fits. One hardware guide concludes that no build of this model actually fits 128GB once you account for context and KV cache on top of the weights. That is a stricter conclusion than the forum posts reach, and it is the one most consistent with the "even the DGX Spark offloads" finding. Which is really the theme: at 128GB you are not escaping offloading, you are only choosing how much of it you do.


Two buyers the cheap route doesn't work for

You're serving more than yourself. Concurrency changes the maths completely. Four R9700 cards are reported serving eight clients at 165 tok/s aggregate; the DGX Spark holds ~26 tok/s per stream with four running. A single 12GB card with experts paging out of system RAM is a one-person machine, full stop.

You barely use it. RunPod rents an RTX 6000 with 96GB of real VRAM at $2.09/hour, so a $3,099 Strix Halo is about 1,483 rented hours — roughly a year at 4 h/day. Alibaba's hosted Flash-Next is around $0.47 per million output tokens, so $3,000 buys about 6.6 billion output tokens, which at 50 tok/s is several years of continuous generation. If the box is going to idle five days a week, rent the bit instead of buying it. The case for owning it is privacy, latency and not having a meter running — not unit economics.


How I'd spend it

Ranked, with the reasoning attached, and all of it conditional on the caveats above.

  1. Own a 12GB card or better? Buy RAM, not a box. 64GB of DDR5 and the 3-bit build in an offloading engine. Spend $880 and find out whether 3-bit quality is actually a problem for your work before spending $3,000 to avoid a problem you may not have.
  2. Want 4-bit in one box? Strix Halo, with Linux on it. Cheapest route that genuinely runs the 4-bit build. The OS choice is a capacity decision, not a taste one.
  3. Want silence and polish? Mac Studio. The highest memory bandwidth here at 614 GB/s, and the nicest software story — but raise iogpu.wired_limit_mb on day one, and know that nobody has published prefill numbers for this model on this machine.
  4. DGX Spark only if you need CUDA specifically. Not because CUDA is good, but because something in your stack requires it. Otherwise you're paying a large premium for a prefill lead that current engines have mostly erased.
  5. RTX Spark: wait. Until someone outside the vendor has timed it on this model.
  6. Four cards only if you're serving a team and the power bill can absorb it.

And the rule under all six: never judge any of these machines on its default engine.


The ceiling on the cheap trick

One honest limitation, because it's the strongest argument for buying the big pool.

The offloading approach is tuned around one model family, and its gains come directly from Flash-Next's specific design — small active weight set, lookup table built to be offloaded. Point the same hardware at a different large model and you're back to whatever a general-purpose runtime does with it. Context is tighter too: one owner running 3.5-bit on two 3090s with 128GB of system RAM reports topping out around 130,000 tokens, against 256,000 on a smaller dense model.

And the next model might not cooperate at all. Qwen describes Flash-Next as a preview of the Qwen4 architecture, and the flash models have grown with every release. The cheap route is a bet that architectural trends keep favouring sparsity and offloading. The big memory pool is not a bet — it loads whatever ships next, in whatever shape, as long as someone writes an engine for it. That's what the premium buys: not speed, and not quality, but not having to care what the next model looks like.


The short version

  • 128GB is one model's 4-bit file size, rounded up to a marketing number.
  • In a sparse MoE, capacity and bandwidth are separate purchases. 6B active parameters out of 125B means the per-token work is small even though the file is huge.
  • Expert offloading makes capacity cheap — hot path on the GPU, dormant experts in system RAM.
  • The number on the box is a ceiling. ~96GB GPU-addressable on Windows Strix Halo and on a 128GB Mac by default; ~122GB on a DGX Spark. The cheapest 4-bit build is 97.7GB before context.
  • The engine is worth more than the hardware brand. 113 → 1,227 tok/s prefill on identical hardware.
  • Prefill and decode are different numbers. Headlines quote decode; long-context agent work lives on prefill.
  • You are buying the fourth bit, and it costs about $700 on a new build or $2,200 as an upgrade.
  • The quality question is unanswered. Nobody has published 3-bit vs 4-bit on a long agent session. Until someone does, test the cheap path on hardware you already own.

If you have one of these running, the thing I'd genuinely want to know is which engine you landed on and what your prefill looks like at depth — that's the number that seems to separate the people who are happy from the people who are still shopping.

Read next

All AI →