The Hard Ones Don't Announce Themselves

Part one worked out whether a model belongs on hardware you own and landed on a routing line.¹ Part two worked out what to configure once it's there, and landed on the same routing line.² Both pieces did the same slightly dishonest thing at the end: gestured at a boundary, called it the real answer, and stopped.

So build it. You have a local endpoint — in my case llama-server on 127.0.0.1:8080, holding Qwen3.8-27B at about 39.6 tokens per second — and you have an account with someone who will sell you a frontier model by the token. Both speak the same API. Now write the thing that decides.

It is harder than it looks, and it fails in a specific place.

The boundary isn't a setting. It's a program, and it has its own bugs.

The dispatch fantasy

The obvious design is a classifier. Look at the incoming request, judge how hard it is, send easy work to the local box and hard work to the frontier. Cheap requests stay home, expensive ones go out, and you capture most of the savings with none of the quality loss.

This doesn't work. Judging how hard a task is requires most of the work of doing it. Not all of it — but enough that the classifier has to be roughly as capable as the model it's protecting you from needing. If you had something cheap that could reliably tell a hard refactor from an easy one, you'd have solved a decent chunk of the refactor.

The proxies people reach for instead are noise dressed as signal. Prompt length doesn't work: a two-line question can require a deep answer and a forty-file context dump can require a rename. Keyword matching doesn't work: "just quickly" appears in front of some of the nastiest requests ever typed. Task-type routing — code to one model, prose to another — is real but coarse, and it splits along a seam that has nothing to do with difficulty.

There's genuine research on learned routers, and the good ones are more subtle than keyword matching — clustering queries and applying cost-aware dispatch before anything runs.³ Prediction still gets you only a useful fraction of the way, because the thing being predicted is not visible in the input.

So it has to be escalation

If you can't tell in advance, run it and find out. Send everything local first. Detect when the local answer isn't good enough. Retry that request against the frontier. This is the cascade, and it's the design nearly every serious system converges on — the original work on it reported cost reductions up to 98% against always calling the best available API, at matched output quality.⁴

The savings are real. But notice what the design just did: it moved the entire problem into one word. Detect.

Everything now rests on a failure signal. Get that right and the cascade is close to free money — you pay local prices for the majority of requests and frontier prices only for the ones that needed it. Get it wrong in one direction and you ship bad output. Get it wrong in the other and you pay for the frontier on every request while also paying for the local attempt you wasted first.

The cascade doesn't solve routing. It relocates the whole problem into the word "detect."

Ask the model? No.

The tempting shortcut is to ask. Have the local model rate its own confidence, and escalate below a threshold. It costs almost nothing and it feels principled.

It doesn't work, and the numbers are worse than most people expect. Across tested models, mean self-reported confidence runs about 72.5% on answers that were correct and 69.4% on answers that were wrong.⁵ Three points. That is the entire signal you're proposing to route production traffic on. On lower-tier models the gap collapses to a tenth of a point — 79.6% against 79.5% — which is to say no signal at all, delivered with total assurance.

The failure has structure. Confidence tracks familiarity rather than correctness: models appear to draw certainty from having seen something like this before, not from having reasoned about this instance.⁶ So confidence is highest exactly where memorization is doing the work, and the cases where it should be lowest — genuinely novel problems — are the ones it can't distinguish from the ones it has effectively memorized.

This is the thing people underestimate about LLM failure generally, and it is fatal here. A wrong answer does not look like a wrong answer. There is no exception, no stack trace, no null. A confidently incorrect refactor is byte-for-byte as plausible as a correct one until something external checks it. Software engineering has spent fifty years building intuitions around systems that fail loudly, and this one fails silently and fluently.

Self-correction doesn't rescue it either. A model asked to review its own output brings the same blind spots to the review that produced the error.

What you can actually detect

Strip out everything unreliable and you're left with a short list. These are the signals that are genuinely external to the model — things the world tells you, not things the model tells you about itself:

Read that table as a cliff rather than a list. Everything above the last row is mechanical, fast, and honest. The last row is where the routing problem goes to die — and for a great deal of what people actually ask models to do, the last row is the only row available.

The sophisticated middle ground is to build calibration you can trust rather than taking the model's word for it — mapping token-level uncertainty to an actual error probability and setting the escalation threshold by cost minimization. Recent work does exactly this and beats entropy thresholding and learned-threshold approaches at the same operating point.⁷ It's the right direction. It also requires per-workload threshold tuning, which means it isn't something you drop in on a Tuesday — you build it for your traffic, and you rebuild it when your traffic changes.

The verifier draws the line

Which gives the thesis, and it isn't a statement about models at all:

Your routing line is drawn by what you can cheaply verify. Not by which models you have.

Two teams with identical hardware and identical API keys will end up with completely different local/frontier splits, determined entirely by how much of their work has a mechanical check attached to it. The one with fast tests and strict types can route aggressively to local, because every wrong answer surfaces in seconds and costs one retry. The one shipping unverifiable output has to send nearly everything to the frontier, because it has no way to find out it was wrong.

The corollary inverts the usual intuition. Agentic coding is the best case for local inference, not the worst. That runs against part one of this series, which routed agents to the cloud on context grounds, and against the general sense that agents are the demanding workload you buy the good model for. But agent loops in a code repository are drowning in free verifiers: every tool call either parses or doesn't, every patch either applies or doesn't, every build either goes green or doesn't. The loop that people describe as the hardest thing to run locally is the one where you can actually tell when the local model has failed you.

And the mirror case is the one nobody flags. "Summarise this document." "Draft the announcement." "What should we do about this customer?" No parser, no compiler, no test. Fluent, confident, unverifiable — and therefore unroutable, because the escalation trigger doesn't exist. You either send it to the frontier or you accept whatever comes back.

The work that's easiest to keep at home is the work with a build step.

"OpenAI-compatible" is a wire format, not a contract

The other half of building this is less philosophical and considerably more annoying. Both endpoints accept the same JSON. They do not behave the same way, and a router written on the assumption that you can swap a base_url will ship bugs that look like model quality problems.

Concretely, from standing this up:

• Reasoning arrives in a different field. Qwen3.8-27B reasons by default, and llama.cpp puts that in message.reasoning_content, separate from message.content. My first test looked like a truncated failure — a 200-token budget consumed entirely by reasoning, content containing a bare function signature. A router that reads content, finds it thin, and escalates would be escalating correct answers because it was reading the wrong field.

• Sampling parameters are mode-dependent and the wrong ones are actively harmful. Thinking mode wants temperature 1.0, top-p 0.95, presence penalty 0.0. Non-thinking wants 0.7, 0.80, and 1.5. These aren't preferences to interpolate between — running the non-thinking presence penalty against reasoning output degrades it, and a router that carries one sampling profile across both endpoints is quietly damaging one of them.

• Tool calling needs the server configured for it. Without --jinja, llama-server doesn't apply the chat template that makes tool calls parse. The failure isn't an error; it's tool calls that don't fire. On the cascade, that reads as local incompetence and triggers escalation on every tool-using request — an infinite bill from a missing flag.

• Advertised features may not be present. The model ships multi-token-prediction weights for speculative decoding; llama.cpp logs unused tensor blk.64.nextn.* and ignores them. Anything in your cost model assuming MTP throughput is assuming a feature this stack doesn't have.

None of these are hard once you know. All of them present as the local model is bad, which is the single most expensive misdiagnosis available to you — because the fix looks like "escalate more," and escalating more is how the economics die.

Every integration bug on the local side masquerades as a quality problem, and the remedy for a quality problem is to stop using the thing you bought.

The asymmetry that eats part one

The costs of a wrong route are lopsided. Send an easy request to the frontier and you've wasted a few cents. Send a hard one to local and you pay the local attempt, the detection, the frontier retry, and the latency of all three stacked in series — on an interactive request, that's the difference between a pause and an abandonment.

Any engineer tuning thresholds against that asymmetry will tune conservatively. Of course they will; the visible failure is the slow wrong answer, and the invisible one is the marginal API spend. So routers drift toward escalation, and the drift is one-directional because nothing pushes back.

There are really two rates here, and conflating them hides the damage.

1. Escalation — the share of local attempts that fail verification and get retried — adds cost but removes no local hours, because you already paid the attempt.
2. Bypass — the share sent straight to the frontier without trying local at all — is the conservative drift proper, and it is the one that empties the machine.

Both rates depend on how often a 27B clears the bar, and that varies by task far more than one number can carry. Qwen3.8-27B sits eighth of twenty on SWE-Bench Pro at 61.7% and eighteenth of twenty-five on Terminal-Bench 2.1 at 73.0%, while leading the office and instruction-following rows.⁸ A single escalation rate across that spread is a fiction, which is why the calculator takes it as an input instead of shipping a default.

Now put that next to part one's arithmetic. Break-even on a $6,500 workstation landed at roughly 170 GPU-hours per month against Runpod's rate, and north of 300 against the market median.¹ Those numbers assume the machine is used. A conservative router is a machine-utilization problem wearing a quality-assurance costume: every threshold nudged toward safety moves hours off the local box and onto the invoice, and if it moves enough of them, the hardware that part one justified never clears its own break-even.

Which means the router — not the card, not the quant — decides whether parts one and two were good advice. You can buy the right hardware, configure it correctly, and still end up with an expensive space heater because the thing dispatching work to it doesn't trust it, and never had a way to find out whether it should.

The part that's half-built

The routing layer is mature. LiteLLM, RouteLLM, the vLLM semantic router and a half-dozen commercial gateways will all classify a request and dispatch it, which is the design this article spent its first section arguing can't work reliably. Escalation ships too: NVIDIA's NeMo Switchyard has an escalation router that starts a session on the weak model and upgrades it under sustained trouble.⁹

But look at what triggers it. Switchyard escalates on an LLM judge reading the conversation transcript, latching to the strong model after a couple of consecutive escalate verdicts. No compile check, no test execution, no external validator. The one productized escalation router in the field picked precisely the signal this article spent a section rejecting — a model's assessment of a model.

The verifier-driven version exists, and it works. MERA runs generated code in an isolated subprocess against test programs, checks schema validity and tool-call legality, and falls back to the larger model when verification fails — reporting 88.3% pass at 60.8% of always-large-model cost.¹⁰ R2V trains small models to recognize when to ask for help.¹¹ Both released code. Both verify against benchmark-provided test programs. Point either at your repository, and there is nothing for it to run.

And the last piece is the one that makes this feel close rather than distant. Coding agents already have the verification loop — Cline will read a failing test, patch the component, run the test command, see a TypeScript error, and fix it. That whole cycle exists and runs today. It just retries the same model every time, because nobody wired a model-escalation policy onto a loop that was already checking its own work.

So the gap isn't conceptual, and it isn't a research problem.

Three pieces exist separately:

• routers that dispatch
• escalators that use the wrong signal
• agents that run the right verifiers without escalating.

Nothing connects your repository's actual toolchain — your test command, your typechecker, your linter — to a decision about which model should have handled the turn.

That's where the local-inference case lives or dies. Not in the quantization thread, not in the break-even spreadsheet, but in the shortest piece of missing software in this entire series: the policy layer on a loop that is already, right now, checking the work.

Build that badly and everything upstream of it was decoration.