anvil-serving: Quality-Gated Local Model Routing

Problem

The result that changed the product was not a crash. It was worse than a crash.

I took Anvil’s real PRD-to-tasks planning prompt, sent it through two local models and a frontier model, then put the outputs in front of a blind judge. The local outputs were structurally valid at least 92 percent of the time. Clean JSON. Parseable task lists. No dangling references that a schema check would reject. Then the quality scores landed: frontier at 24.75 out of 25, fast local at 16.0, heavy local at 13.25.

The gap was dependency ordering. Frontier got it right. Local mostly did not.

That is the failure mode anvil-serving exists to catch. A coding agent plan with tasks in the wrong order does not necessarily fail when it is generated. It fails later, after the harness has already trusted it, claimed work, edited files, and spent context. A generic proxy cannot see that. A generic proxy can move tokens, route aliases, and retry on transport errors. It does not know whether local can do this class of work on this operator’s workload.

Anvil and anvil-serving are adjacent, but they are not the same layer. Anvil is the durable system of record: requirements, tasks, claims, leases, evidence. anvil-serving sits in front of coding harnesses and answers a different question: which model should be allowed to do this work right now?

Approach

Intent Travels Through the Model Field

The awkward constraint became the API. Most coding harnesses already forward a model field. They do not all forward custom routing metadata, and they do not agree on a native intent schema. So anvil-serving treats the model field as an intent channel.

Instead of asking for qwen35-awq-local or gpt-oss-20b, a caller can ask for:

planning
quick-edit
review
chat
long-context

Those presets can be bare or namespaced as anvil/planning. The router expands the preset into constraints: context length, privacy posture, structured-output needs, tool support, and cost ceiling. A direct model pin still exists for reproduction and debugging, but the normal surface is intent. The caller says what kind of work this is. The router owns which backend is currently trusted for it.

That choice matters because models churn. Hardware changes. A quant swap can invalidate yesterday’s confidence. The preset vocabulary stays stable while the tier behind it can move from fast-local to heavy-local to cloud based on measurement.

Filter, Then Rank

The routing rule is deliberately boring: filter by hard constraints first, then rank by the quality profile. A model that cannot fit the context, violates a privacy requirement, or lacks the needed wire behavior is out before quality scoring starts.

The quality profile is the product. It is keyed by model and work class, with decisions like allow, allow-with-verify, and deny. Planning is the clean example. The hard eval says local was structurally valid but weak on dependency reasoning, so planning stays cloud-default until fresh measurement proves otherwise. That is not pessimism. It is engineering discipline.

Here is the shape of the operator contract:

[router]
profile_path = "./profile.candidate.json"
exhaustion_status = 503

[router.metered_cloud]
planning = true
review = true

[[tiers]]
id = "fast-local"
privacy = "local"
provider = "openai-compatible"
base_url = "http://127.0.0.1:30001/v1"

[[tiers]]
id = "heavy-local"
privacy = "local"
provider = "openai-compatible"
base_url = "http://127.0.0.1:30000/v1"

The important part is what is not implicit. Metered cloud is opt-in by work class. A local miss does not silently spend against a cloud key unless the operator configured that class to allow it.

Verification Happens Before Trust

Routing ahead of time catches the known bad classes. Verification catches the cheap obvious failures that still happen in real traffic.

Local responses pass through gates for empty output, truncation, malformed tool-call JSON, invalid code shape, broken diffs, and refusal markers. The depth depends on the work class and tier. If a local response fails, the router walks the tier ladder: fast-local, heavy-local, then cloud if explicitly configured.

Streaming made this harder than I wanted. If a local tier starts streaming bad output, the harness has already seen it. You cannot unstream a bad token. So verify-gated local paths use a commit window: buffer enough to verify before the first byte leaves the router. Cloud can still stream true upstream SSE when it is the trusted tier. Local pays a little first-token latency where the evidence says it should.

That is the tradeoff I accepted. I would rather delay the first token than let a failed local answer leak into an agent run.

The Serving Layer Is Part of the Product

A router with opinions about quality needs a serving substrate it can inspect and operate. anvil-serving includes the substrate commands because “just point it at a model” was not enough.

models sync builds a local model catalog. deploy and init generate consistent configs. preflight checks correctness against a serving endpoint. benchmark measures capacity. multiplexer swaps models on one GPU while draining in-flight requests. Later releases added router and serve lifecycle commands, harness sync for OpenClaw, and host management for WSL and Docker Desktop incidents that were painful enough to turn into code.

The request path itself stays intentionally small. The Python package has zero required runtime dependencies. The hot path uses stdlib HTTP machinery instead of a web framework. That is not a universal recommendation. It is a tradeoff for this project: fewer moving pieces in the part that has to decide whether a model answer can be trusted.

What I Learned

Structurally valid is not the same as good. This is the lesson that keeps showing up in agent infrastructure. A schema can tell you the JSON parsed. It cannot tell you that the dependency direction is wrong. In long agent runs, the semantically wrong answer is more dangerous than the malformed one because it looks safe enough to continue.

The integration point is the harness, not the ledger. I wanted the Anvil relationship to be tighter at first. That instinct was wrong. Anvil keeps the record of work. The model call goes through Claude Code, Codex, Aider, OpenClaw, or another harness. If the router does not sit where the model traffic already flows, it becomes another beautiful internal abstraction that the actual request never touches.

Local-first needs humility to be useful. I want local models to carry more of my coding workload. That does not mean pretending they are ready for every class. The honest version is narrower and stronger: local where it has been proven, cloud where it has not, verified at the boundary. That posture makes local useful because it stops asking local to be magical.

Defaults are a trust contract. The local-only default matters. If a router silently spends cloud tokens every time local misses, it is not local-first. It is a cloud proxy with local optimism. anvil-serving’s default exhaustion response is intentionally blunt: no local tier cleared the gate, and no metered cloud path was authorized. The operator decides where money can be spent.

The transferable principle is simple: own the measurement, not the model. Models will keep changing under the system. That is fine. The durable thing is knowing what each one has earned the right to do, checking that trust at the boundary, and refusing to guess when the data is not there.