I have been constantly at odds with the age-old debate over the local-versus-cloud decision. Above roughly 100B active or dense parameters, the cloud wins. Below 70B, local wins. The narrow band in between is where the argument actually happens. Let's dig into this further and, as a bonus, check out the interactive app below to make it easier than just reading an article.
The exact line depends more on quantized weight size and active parameter count than on the marketing headline. A 30B MoE model with 3B active is not the same problem as a 30B dense model, even when both fit in the same VRAM budget.
That distinction used to be a caveat. It is now the structure, especially since the local models have finally caught up to the frontier-level class of workloads. Total and active parameters have fully decoupled: DeepSeek V4 Flash is 284B total and 13B active, V4 Pro is 1.6T total and 49B active, and Qwen3.8-Max is 2.4T total with 95B active. Research models are reaching strong results at 35B total and 3B active.1 As a result, the headline parameter count no longer tells you what hardware you need, because it answers only half the question.
There are two questions, and they load different parts of the machine.
- Does it fit is a capacity question — total parameters and quantization against memory size.
- Does it run is a bandwidth question — active parameters against memory throughput.
A model can pass one and fail the other, and which one it fails tells you whether you need a bigger machine or a faster one.
The GDDR7 shortage has repriced consumer and workstation GPUs upward — memory is now more than 80% of a card's bill of materials — while cloud GPU rates have fallen roughly a third. Local got more expensive. Cloud got cheaper. Those are not independent trends around a fixed crossover; they are both blades of the same scissors, closing in the same direction. Any break-even you calculated in 2025 is wrong now, and wrong in a predictable direction.
The structural argument below — where the breakpoints sit, why context is the hidden half of the VRAM equation, why workload shape matters more than model size — holds, with one caveat I will come back to: the context half is the part being engineered away fastest. The dollar figures are all as of August 2026 and should be re-checked before you spend anything.
| Model class | Active | Q4 weights | Fits on | Runs well on |
|---|---|---|---|---|
| 7–14B dense | all | 5–10 GB | 16 GB GPU | anything that holds it |
| 20–32B dense | all | 12–24 GB | 24–32 GB GPU | anything that holds it |
| 40–70B dense | all | 25–45 GB | 32–48 GB GPU | high-bandwidth GPU |
| 70–120B dense | all | 45–80 GB | 2× consumer GPU or 96 GB pro GPU | high-bandwidth GPU only |
| 30B MoE | 3B | ~18 GB | 24 GB GPU | almost anything, unified memory included |
| 284B MoE | 13B | ~145 GB | 256 GB unified memory or multi-GPU | unified memory is sufficient |
| 1.6T MoE | 49B | ~800 GB | datacenter topology | datacenter |
| Kimi K3, 2.8T total | unconfirmed | ~649 GB at extreme quantization | 4× B200-class system | datacenter |
Read the sparse rows against the dense ones. A 284B model with 13B active needs roughly three times the memory of a dense 70B and runs several times faster on the same hardware, because each token only touches 13B of weights. Sparsity buys speed with capacity. That is exactly the trade memory-rich, bandwidth-poor hardware is built to take, and it is why the old habit of sorting models by headline size now produces inverted answers.
The 5090 case
The RTX 5090 has 32 GB of VRAM and launched at $1,999. That number is now fiction, and I still kick myself for not buying when I saw the writing on the wall. Perhaps I was hoping the big guys would solve more of the equation before it hit my wallet.
Street pricing in August 2026 starts around $2,900 for mid-range cards, sits at roughly $4,300 on Amazon, and runs past $5,000 for premium and liquid-cooled variants. Used cards clear near $4,000.2 The card is selling for more than double its launch MSRP, and the driver is the memory shortage rather than anything about the silicon.
Warning
Quote the MSRP in this market and every number downstream of it is wrong... it is now a distant memory of days long gone.
A workstation built today lands closer to $6,500 once you account for power, cooling, and everything else. That machine runs 7–32B models comfortably, some 40B-class models, 70B models at aggressive quantization or with CPU offload, and most MoE models whose stored weights fit within 32 GB.
On a dense 32B model that gap is the whole story — the 5090 finishes about three times sooner, and you feel every second of it. On a sparse MoE model of the same stored size, where each token touches a fraction of the weights, both machines clear conversational speed with room to spare, and a difference you cannot feel is not worth five thousand dollars. Same two cards, opposite conclusions, decided entirely by which axis the model loads.
Runpod lists an RTX 5090 at approximately $0.99/hour on demand.4
The math:
- $6,500 workstation
- $1,200 residual after three years
- $0.10–$0.15/hour electricity under inference load
- $0.99/hour cloud equivalent
Important
Break-even lands at roughly 6,100 hours over three years. About 170 GPU-hours per month.
Under 100 hours/month, rent. Around 170, indifferent. Above 250, ownership wins. At continuous operation, ownership still wins decisively.
At the old MSRP, those thresholds were 50, 105, and 150. The shortage moved every one of them up by more than half, and it moved them in the same quarter that cloud rates fell.
Two corrections cut against each other.
Development, compilation, image and video generation all consume hours the pure inference math ignores, which pushes the effective break-even down. But the residual assumption is now conservative in a way it has never been before: a GPU whose used price exceeds its launch MSRP is not depreciating like a tool. If cards hold value through the shortage, the $1,200 residual is too low, and ownership looks better than the table says.
That is a strange thing to have to write about a consumer graphics card, but it is also the clearest signal available that this market is not pricing compute normally.
The other side moved too. Spot instances run as low as $0.09/hour, and the median on-demand rate across providers fell roughly 34% in the year to August 2026 — from about $0.88 to $0.58.5 Run the break-even against a $0.58 median instead of Runpod's $0.99, and ownership needs well over 300 hours a month to justify itself.
The scissors close from both directions. That is the finding.
The context tax
The parameter count is half the VRAM equation. The KV cache is the other half.
Every token in context requires KV cache memory. Size scales with layer count, hidden dimension, and KV head count under GQA. For a 70B model at Q4, full 128K context can add another 20–40 GB on top of the ~40 GB in weights. Your 32 GB card is out of the running before you finish loading.
For devs, this reshapes the decision in three ways.
-
Effective context is smaller than advertised. A 32B model that "supports 128K" runs out of VRAM budget on a 5090 well before you hit the ceiling. In practice, you get 32–64K comfortably, and pay for the rest in offload or KV-cache quantization damage.
-
Agentic workflows shift the profile. Tool-heavy loops accumulate context fast — MCP responses, file reads, search results, prior turn outputs. A Claude Code session or long-running agent crosses 100K tokens in a working day without trying. That workload is a bad fit for a 5090 running a local model. It is a good fit for a cloud endpoint with prompt caching, where repeated prefix becomes nearly free on the second call.
-
Throughput matters more when the loop is tight. Agents care about time-to-first-token and steady tokens per second under multi-turn load. Local inference on long context degrades on both — prefill is slow, and generation slows as the cache grows. A datacenter GPU with cheap prefill and a large KV cache is a fundamentally different machine.
Tip
The practical rule: run 20–40B models locally for one-shot and short-context work. Run agents against cloud endpoints where prompt caching and long-context prefill are priced sanely — or against local unified memory, if you have bought enough of it. The local-cloud split is not just by model size. It is by workload shape.
One caveat: prices are not the only thing that can move a structural argument; algorithms can too, and the KV cache is under direct attack. Published work now reports physical KV-cache reductions above 90% at 500K context with accuracy preserved.6 Weights are shrinking slowly and cache is shrinking fast, which makes the context tax the number here most likely to be wrong first — and it will be wrong in the direction that favors local.
The awkward tier: 70–120B
This is where local economics deteriorate — or did, until a third kind of machine arrived.
A 70B model at Q4 runs around 40–45 GB before context and runtime overhead. That does not fit cleanly on a 32 GB 5090.
Three options exist. Two are uncomfortable in the same way; the third is uncomfortable in a different one.
-
Two RTX 5090s now cost $11,000–$14,000 as a complete build. You get 64 GB aggregate VRAM, but consumer Blackwell cards do not have datacenter-class GPU-to-GPU interconnect. Tensor-parallel inference crosses PCIe, and scaling is not linear. This makes sense when you run 70B models frequently, use the GPUs hundreds of hours monthly, and accept dual-GPU workstation complexity. It is not a path to 200B or trillion-parameter models. You are building a server one consumer card at a time — and the shortage has roughly doubled the cost of that strategy.
-
An RTX PRO 6000 Blackwell has 96 GB GDDR7 ECC and lists at $13,250 for the GPU alone. A complete workstation approaches $17,000. This is an excellent card and a weak value proposition for an individual: roughly two and a half times the price of a 5090 workstation for three times the memory — not two and a half times the inference speed.
The trajectory matters more than the sticker. That card launched in March 2025 at $8,565. It is up 55% in sixteen months, on the same 96 GB clamshell GDDR7 design that makes it acutely exposed to the memory market.7 Nothing about the product improved. The memory got scarce.
When the hardware you would buy to escape cloud dependence is itself appreciating at 55% a year, that escape is not working.
The third option refuses to compete on bandwidth at all, and solves the capacity problem for a fraction of the money.
- A DGX Spark holds 128 GB of unified memory at $4,699 — itself up about $700 from a $3,999 launch, because nothing escapes this market. Two of them give 256 GB for roughly $9,400 and draw around 250W combined, against $11,000–$14,000 for dual 5090s at 64 GB, or roughly $17,000 for a 96 GB pro workstation. AMD's Strix Halo reaches the same 128 GB from about $2,000.8 On capacity per dollar the argument is over before it starts.
The catch is the specification most summaries skip. A Spark runs at 273 GB/s of memory bandwidth and the AMD box at 256. A 5090 is near 1.8 TB/s, an M3 Ultra 819 GB/s. On a dense model that gap is disqualifying: a 70B at Q4 has to stream roughly 35 GB per token, and at this bandwidth you get single-digit tokens per second. The machine that fixes the awkward tier is bad at the awkward tier.
Sparsity is what rescues it. Running DeepSeek V4 Flash across two Sparks, each token touches 13B of active weights instead of 284B, and published serving recipes report roughly 35–45 tok/s on mixed content, 60–67 on favorable code generation with speculative decoding, and around 115 tok/s aggregate across two concurrent sessions at a 1M-token context window.9
Hold that against the dense rows above. The same pair of boxes that would crawl on a 70B runs a 284B model at working speed. The hardware did not get better. The model got shaped to fit it.
So the awkward tier has an answer, on a condition: buy capacity at low bandwidth, and you own a machine that is excellent on sparse MoE models and mediocre on everything else. That is a bet on the direction of model architecture, layered on top of the bet on memory prices. Cloud still wins cleanly for dense models in this range, and for anything where single-stream latency is the thing you are buying.
The Mac exception, which just got smaller
A high-memory Mac Studio changes the capacity equation without changing the performance equation.
The M3 Ultra delivers approximately 819 GB/s of memory bandwidth across a 32-core CPU and 80-core GPU. It cannot match B200 inference speed. It is a huge-memory inference appliance, not a compressed datacenter — but for loading weights that would otherwise demand five 96 GB professional GPUs or several H100s, it was the only sane single-box answer.
Was. The 512 GB configuration has been retired. Apple now tops the Mac Studio out at 256 GB, alongside a general repricing of memory upgrades.10
That is the same shortage, arriving through a different vendor. Apple is buying from the same constrained memory market as everyone else, and unified memory is not exempt. The 512 GB machine — when you could still order one — ran to roughly $14,100 fully configured.
So the capacity argument halves. A 256 GB ceiling covers models up to roughly 200 GB of weights with context headroom, not the 100–400 GB range this section was originally written around. Everything above that is now cloud, a used 512 GB machine, or a multi-GPU server.
Note
The Mac exception is still real inside its new ceiling. It becomes rational when you need up to ~200 GB of weights loaded regularly, batch-one interactive performance is sufficient, privacy or offline availability matters, and you accept low-double-digit tokens per second on enormous models.
Kimi K3 at ~649 GB was already out of reach for a 512 GB Mac without storage offload.11 At 256 GB it is not a conversation. At that point, performance is theoretical.
The architecture
The recommendation splits by workload shape before it splits by model size, because the two local machines worth buying are good at opposite things.
Buy bandwidth for interactive, short-context work: one RTX 5090 workstation, running Qwen-class 14B/32B, coding models in the 20–35B range, 30B MoE models, 70B when a good quantization fits, and local image and short-video models. This is still the high-value tier for most people. A 32B model running rapidly and continuously produces more real work than occasional access to a frontier-sized model.
Buy capacity for agents: concurrent sessions, long context, and models no single consumer card can hold. That means 256 GB of unified memory — two Sparks, or a 256 GB Mac Studio — running sparse MoE models, where aggregate throughput across several agents matters more than how fast any one chat finishes. This is the machine the context-tax section was pointing at, and it is the machine the 5090 cannot be.
The Mac Studio is the quieter version of that second option, chosen for capacity, silence, unified memory, and persistence rather than throughput. If your workflow genuinely needed the retired 512 GB part, that is now a used-market hunt or a cloud decision.
Keep in cloud: anything requiring more than 96–128 GB of fast VRAM for good performance, dense models needing three or more discrete GPUs, H100/H200/B200-optimized workloads, Kimi K3, infrequently used frontier-sized models, and any workload where you want to change hardware generations frequently.
The breakpoints in plain terms:
- Up to 32B dense: local, unquestionably.
- 40–70B dense: local on one high-bandwidth consumer card when it fits.
- 70–120B dense: the financial crossover. Cloud usually wins under 200–300 hours monthly.
- Sparse MoE up to roughly 250 GB of weights: local on unified memory, provided the active count stays small.
- Above 120B dense, or anything needing three or more discrete GPUs: cloud by default.
- Kimi K3 scale: cloud, no argument.
The right shape is horizontal local intelligence with vertical cloud escalation. Run a strong 20–40B model on a 5090 for routine work, or a sparse MoE model on unified memory if the work is agentic. Route the genuinely difficult prompts — and anything that needs the frontier — to Kimi K3 or another cloud model where prompt caching makes the KV cache someone else's problem.
That captures the privacy and marginal-cost wins without buying hardware sized for the rarest workload or the longest context.
What actually changed
The breakpoints in this piece are structural. Where a model fits is a function of quantized weight size, KV cache, and bandwidth, and none of that cares what a card costs.
The economics are not structural, and they moved hard in one direction.
A year ago the case for local inference was a straightforward capital argument: buy the card, clear break-even in three or four months of serious use, own your marginal cost afterward. That argument now needs 170 hours a month against Runpod's rate and over 300 against the market median — while the machine that used to be the capacity escape hatch lost half its memory ceiling and the workstation card that replaced it went up 55%.
The memory shortage did something more interesting than raise prices. It made owning compute behave like taking a position in a commodity, at exactly the moment renting it got a third cheaper.
It also obscured a second axis. Capacity ages well; bandwidth ages poorly. A 256 GB machine bought today will still hold next year's models, because architecture is moving toward more total parameters and fewer active ones — the direction that rewards capacity. What will look dated is the 273 GB/s, or the 819, long before the memory fills up.
So the position is more specific than "compute." You are long memory capacity and short memory bandwidth, and the trade only works if the open-weight ecosystem keeps producing models shaped for that tier. On current evidence, it will: leading open-weight models are running months rather than years behind the closed frontier, and the 100–300 GB local tier now has enough of a market to be targeted deliberately.12
Local inference did not stop making sense. It stopped being the default, and it became a bet — on where memory prices go, and on models staying sparse.
Anyone telling you otherwise... is likely trying to sell you something.