Structurally Valid Is Not Good Enough: Why My Model Router Measures Quality

The views expressed here are my own and do not represent those of any current or former employer.

I took a real planning prompt, the PRD-to-tasks step from my own pipeline, ran it through two local models and a frontier model, and put all three outputs in front of a blind judge. The local outputs were structurally valid at least 92 percent of the time. Clean JSON. Parseable task lists. Everything a schema check would wave through. Then the judge scored them: 24.75 out of 25 for frontier, 16.0 for my fast local model, 13.25 for the heavy one. Local landed at roughly 55 to 65 percent of frontier quality, and the gap concentrated in exactly one skill: dependency ordering. Frontier scored 5 out of 5 on it. Local scored about 2.

That result is the whole product. Structurally valid but semantically wrong is the one failure a static proxy cannot see, and it is the one failure that quietly poisons a long agent run. A plan with tasks in the wrong order does not error. It executes.

So I built anvil-serving: a quality-gated router for coding harnesses that sends work to local models where measurement has proven they can handle it, keeps it on the cloud where they have not, verifies every local response, and falls back automatically when verification fails.

Transport Is a Commodity

Every proxy in this space moves tokens competently. LiteLLM, claude-code-router, Ollama, OpenRouter: all of them route, and all of them route on static rules. A model name. A cost ceiling. A regex over the request. What none of them know is whether local can do this work, on your workload, right now. That knowledge is the entire game.

Layer	Routes on	What it knows about output quality
LiteLLM	model name, cost rules, static config	Nothing
claude-code-router	regex over the request	Nothing
Ollama	whichever model you asked for	Nothing
OpenRouter	price and provider availability	Nothing
anvil-serving	a measured quality profile per model and work-class, then verification of local responses	That knowledge is the product

The defensible asset is not the transport. It is the quality profile, measured per model and per work-class on the operator’s own workload, plus the verify-and-fallback loop wrapped around it. anvil-serving is not a proxy with a fallback bolted on. It is a measurement system with a proxy attached.

Intent Rides in the Model Field

A single operator installs anvil-serving in front of any coding harness: Claude Code, Codex, Aider, pi, Cline, Continue. OpenClaw is the near-first-class beachhead, with a reference adapter plugin. The integration trick is deliberately boring: callers put an intent preset in the model field. planning, quick-edit, review, chat, long-context, bare or namespaced as anvil/planning. The model field is the one routing channel every harness exposes and forwards verbatim, so I made it the API. /v1/models advertises the preset vocabulary, which means the presets show up in the harness’s own model picker. A model: pin remains as the escape hatch for repro and debugging, and when no preset is given, a Tier-0 classifier infers a work-class.

One endpoint published to the harness: intent goes in the model field, the measured profile decides, verification gates what comes back.

Each preset resolves to hard constraints: context length, privacy, tool support, cost ceiling. Those constraints filter the candidate list. Then the measured quality profile ranks the survivors. Filter, then rank, in that order, every time. And work-classes the profile marks deny never reach local at all. On the shipped seed profile, planning never reaches local, period, and multi-file-refactor reaches only the heavy tier, gated behind the full verify chain, because of the numbers at the top of this post. The router that saw local score 2 out of 5 on dependency ordering does not get to feel optimistic about planning.

Verification Gates Everything Coming Back

Routing on measured quality only works if you also check the work. Every local response is verified cheaply, and the depth matches the trust: tiers the profile trusts outright get a minimal gate (non-empty, not truncated), while verify-gated work-classes get the full chain (tool-call JSON valid, code parses, diff well-formed, no refusal markers). Cloud responses pass through untouched; the gate exists for the tiers I have reason to doubt. On failure the router walks up the tier ladder: fast-local, heavy-local, cloud.

The tier ladder: resolve intent, serve from the cheapest trusted tier, verify, escalate on failure.

Streaming is where this usually falls apart, so I built it around a commit window: zero partial tokens from a failing local tier ever reach the client. Trusted cloud tiers stream true upstream SSE with real time-to-first-token; local tiers under the shipped default trade first-token latency for the minimal gate, deliberately. The harness never sees a response that later turned out to be garbage, because garbage gets discarded before a single byte leaves the router.

Walk it yourself. Pick a preset, decide whether local output survives verification, and watch where the request lands.

The tier walk, step by step

Pick an intent preset and decide whether local output survives verification. The walk follows the shipped seed posture: keyless by default, zero metered cloud.

Intent preset quick-edit chat review long-context planning Local output passes verification

Illustrative of the shipped seed profile; your profile is measured on your own workload, so the walk can differ per work-class. The checks shown are the real ones: trusted classes get the minimal gate (non-empty, not truncated), verify-gated classes get the full chain (tool-call JSON valid, code parses, diff well-formed, no refusal markers).

Local-Only Is the Default Posture

The shipped default is local-only with zero metered cloud. When local is exhausted, the router does not silently spend money on my behalf. It returns a clean exhaustion status, 503 by default, and the harness’s own gateway fails over on its own key. I call this advise-and-defer, and it is the same division of labor I described in building a personal AI gateway: the router owns the local decision, the gateway owns the cloud relationship. Metered cloud exists, but it is opt-in per work-class. There is also a decision-only endpoint, POST /v1/route, that returns the routing verdict without generating a token, for callers that want the judgment without the inference.

This is the thesis from volume goes local, stakes go to the cloud turned into shipped code. The high-frequency, low-stakes work flows to hardware I own. The high-stakes reasoning stays where the measured quality lives.

The Substrate Underneath the Router

A router with opinions about quality needs a serving layer it can trust, so anvil-serving ships that too. A single-GPU model multiplexer swaps models on demand and drains in-flight requests before evicting anything. models sync maintains catalog cards of per-model serving facts, which turns the read-the-model-card ritual into a command. deploy and init generate a mutually consistent Docker compose file, serve manifest, and router config from detected hardware. doctor preflights the environment, preflight gates correctness, benchmark measures capacity.

My reference deployment is one box: an RTX PRO 6000 with 96 GB as the heavy tier and an RTX 5090 with 32 GB as the fast tier, no NVLink, an asymmetric pairing behind batching engines (SGLang and vLLM). The router is the only endpoint published, token-authed, containerized.

The engineering posture matches the positioning. v0.7.2, MIT, Python 3.11 and up, and the hot path is stdlib only: http.server and urllib, no FastAPI, no SDK. Yes, I wrote an HTTP service on http.server in 2026, on purpose. Every dependency I did not add is a dependency the request path does not have to trust. What backs that choice is 993 tests on a 6-cell CI matrix, Ubuntu and Windows across Python 3.11 through 3.13.

The Edges I Publish on Purpose

A router whose entire pitch is measured confidence has no business implying measurements it has not made. So the honest edges ship in the docs, deliberately.

Only the planning work-class rests on hard eval data so far. The other promotion verdicts are seeded, pending calibration against real traffic. And even the hard measurement is pinned to the models I served when I ran it; the heavy serve has since moved, so re-running the shadow-eval against the current pair is the next item on the calibration list. I publish all of that because a seeded verdict presented as a measured one is exactly the failure mode this product exists to kill.

Under OpenClaw’s current provider resolution, the keyless 503 handoff is not a reliable safety net for local-preferred classes. That is written up as ADR-0005, with mitigations, including the opt-in metered cloud tier. The limitation is documented where an operator will find it before it finds them.

And the hardening is recent and real, because live incidents made it real. A harness sent max_tokens=1 and got exactly what it asked for, every turn: a 503 each time, until the circuit breaker tripped. Fixed in v0.7.1. Docker bind-mounted weights made cold loads take 20 to 90 minutes on WSL2. Fixed in v0.7.2 by moving weights to a named volume. Neither fix came from a whiteboard. Both came from the router running in front of real work, which is the only calibration source I trust.

Own the Measurement

Anyone can move tokens. Transport was a commodity the day the second proxy shipped. The durable layer is knowing, with numbers from your own workload, what each model can be trusted with, verifying that trust on every response, and refusing to guess when the numbers are not there yet. Models will keep churning under this thing, and that is fine. That is the design. Build the anvil, let the hammers change.