The Model Name Is Just a String

The views expressed here are my own and do not represent those of any current or former employer.

The second GPU changed the question. When the RTX PRO 6000 went in next to my RTX 5090, I had 96 GB of VRAM on one card, 32 GB on the other, and no NVLink between them. I knew I could not split one model across that pair and still call it efficient. Mismatched VRAM with no interconnect is not a pool. It is two engines that happen to share a power supply.

So I did what the topology was asking for. A heavy, capable model pinned to the big card. A fast, small model pinned to the other. Each served behind a batching engine (SGLang or vLLM), so concurrent agent calls batch instead of stalling. I had already made the case for the big card in I bought a GPU to run my own agents. The second card is what turned a workstation into tiers.

The ambition at that point was modest. I built the serving layer so I could connect it to different harnesses and actually use both cards. That is all it was for.

The Model Field Is an Input, Not a Label

Every coding harness I care about speaks the same dialect: an OpenAI-compatible endpoint and a model field that carries a name. Here is the observation that started compounding: the harness does not validate that name against anything. It is just a string, passed through. If it is a string, it is an input. And if it is an input, what is stopping me from routing on it?

Then came the second step, the one that mattered. If I control the resolver, who cares whether the string is the actual name of a model? I can put the intent in it instead: what the person, or the agent, is actually trying to do. “Chat.” “Planning.” The work class becomes the model name, and the name abstracts to whatever we have measured to be the best model for that work. I can swap models underneath and every harness keeps the same configuration.

This is not an exotic idea; it is how I already work in the cloud. In Claude Code I can say: use Sonnet for implementation, Opus for reviews, Haiku for searches, and the workflow just knows. But the moment you enter the world of local models, hardcoding a specific checkpoint name into your processes is how configurations rot. You want the abstraction. You do not staff a team by writing one engineer’s name into the org chart; you write the role, and the person filling it can change. Intents are roles. Model names are people.

So the router grew intent presets: planning, quick-edit, review, chat, long-context. They ride in the model field, bare (planning) or namespaced (anvil/planning), and the vocabulary is advertised on /v1/models, so the presets show up in harness model pickers like any other model. Try the resolution yourself:

What the router does with the model field

Type anything a harness might send as the model name, or pick an example. Presets work bare ("planning") or namespaced ("anvil/planning").

qwen3-coder-next (a literal model name) planning quick-edit review chat long-context

intent preset

Resolves to

Why

On verify failure

Illustrative resolution table. The real router keys every route off a measured per-(model, work-class) quality profile, verifies structure on the way out, and treats metered cloud as opt-in: a local miss returns a clean 503 the harness gateway fails over on.

The Substrate Grew Underneath the Router

Once the router existed, we kept noticing we were rebuilding the same operational pieces around Docker to keep any of it repeatable, so we made them commands. A multiplexer swaps models on one GPU, draining in-flight requests before evicting the old model, because an agent mid-run does not appreciate its backend vanishing. models sync builds catalog cards of per-model serving facts. deploy and init generate correct bring-up (the compose file, the serve manifest, the router config) from detected hardware plus those cards.

That last piece attacks the hardest part of local serving. You find a model. Everyone is talking about it. You go to run it, and it falls over, because the settings it needs were sitting in a model card you did not read. Before, I would go hunting through model cards to figure out how to configure a thing; now it is an actual command. That is the same bet I described in owning my own substrate: own the durable layer, and let the models churn on top of it.

The Measurement That Turned a Proxy Into a Product

Up to here, I had a clever proxy. What changed the product was deciding to measure it. I took a real PRD-to-tasks planning prompt, the kind of structured decomposition my agent workflows actually run, put every tier through it, and scored the outputs with a blind judge.

Tier	Structural validity	Blind judge total (of 25)	Dependency ordering (of 5)
Frontier cloud	clean	24.75	5.0
Fast local	see note	16.0	about 2
Heavy local	see note	13.25	about 2

Structural validity was measured across the local outputs in aggregate: at least 92%, with 5 of 6 outputs scoring 100%. It was not broken out per tier.

Read the columns separately, because that is the finding. The local models produced structurally valid output at least 92% of the time: parses cleanly, no cycles, no dangling edges. If you only checked structure, you would ship it. But the blind judge put local quality at roughly 55 to 65% of frontier, and the gap is not smeared evenly across the rubric. It is concentrated in dependency-ordering reasoning, the part of planning that decides what must happen before what. Frontier scored 5.0 out of 5 there. Local scored about 2.

Structurally valid but semantically wrong is the precise failure a dumb proxy feeds silently into a long agent run. Nothing errors. Nothing retries. Hours later you are debugging an agent that faithfully executed a plan whose ordering was wrong from minute one.

I want to be honest about what those numbers say, and I mean this as conviction, not hedging: on my hardware, when I measured, local models were not close to frontier on planning. Those heavy-tier numbers are pinned to the model I served at the time; the heavy serve has since moved, and re-running the eval against the current pair is the next measurement on my list. That gap is not a reason to unplug the cards. It is the reason the router exists. A router that pretends the gap is not there is worse than no router at all.

Routing by Name Is Routing Blind

The consequence falls straight out of the table. You cannot route by model name, because the name says nothing about fitness for a work class. You cannot route by cost, because cheap and wrong is expensive. You cannot route by regex on the prompt, because “make a plan” and “rename this variable” can look similar at the surface. The only defensible routing key is a measured quality profile per (model, work class), backed by cheap structural verification on the way out, with automatic fallback up the tier ladder: fast local, heavy local, cloud. Local where it has been proven. Cloud where it has not.

And the default posture stays local-only, with zero metered cloud. When every proven local tier misses, the router does not quietly call an API with my wallet attached; it returns a clean exhaustion signal (a 503 by default) that the harness’s own gateway fails over on, using whatever cloud model the harness already trusts. Metered cloud through the router is opt-in, deliberately.

All of this shipped as anvil-serving, the quality-gated local-model router for coding harnesses: v0.7.2, MIT licensed, stdlib-only Python, 993 tests. (993 tests is a lot of tests for “point the harness at a different port.” That is the tell that it stopped being a port.)

The Measurement Is the Product

The transport was never the hard part. Anyone can stand up a proxy that forwards OpenAI-compatible traffic; that is a weekend. What compounds is the other two moves. Name the work, not the model, so every harness and workflow stays stable while the models underneath churn. Then gate every route with a measurement you ran yourself, on your prompts, on your hardware, because structure is checkable by machine and quality is not, and the space between those two checks is exactly where silent failures live.

The model name is just a string. What you resolve it with is your judgment, and judgment only counts if you measured it.