What Wins in a Benchmark Is Not What You Ship

The views expressed here are my own and do not represent those of any current or former employer.

I almost promoted a model that crashed twice the moment real work touched it. Not a model that failed a test. A model that passed everything I threw at it in the lab, and then died under a single normal request from an agent that was already running. The request that killed it asked for only 5,120 output tokens, a tiny ask by the standards of what I was testing.

This is the post about that gap, and about the rule I built my serving layer around because of it: a benchmark never promotes a model. A person authorizes the promotion, and only after real client traffic has been given the chance to break it.

The Profile That Looked Like the Winner

The context pain I was trying to solve was real. Agents on a 96 GB card felt cramped, so I moved to a matched pair of RTX PRO 6000 cards running in exclusive tensor-parallel mode, and ported the community DeepSeek V4 Flash 0731 recipe to that topology. That gave me three profiles against the same pinned model, engine, quantization, and two-card arrangement: 128K, 650K, and 1M.

I held everything fixed except two knobs that trade against each other: how many sequences the server admits, and how many tokens it batches in a step. Growing the context window means shrinking the batching budget. That trade is what let 650K and 1M exist in the first place.

Profile	Batching budget	Needle retrieval	What the gates showed
128K	4,096 batched	baseline	stable, the known-good floor
650K	4,096 batched	~640K recovered	passed protocol + matched performance
1M	2,048 batched	~985K recovered	passed synthetic, matched perf, post-stress smoke

Read the last row the way I should have read it the first time. The 1M profile recovered a needle near 985,000 tokens. It passed the protocol gate, it matched the 32K performance run, and it came through a post-stress smoke. On a spreadsheet, 1M was the more useful result. That is exactly why it was tried first as the client-facing configuration, rather than rejected from an estimate.

The Number That Was Not Tested

Here is what the synthetic gates did not measure: the shapes of prompts and tool calls that a real coding agent produces. Near-limit retrieval tells you the model can find a needle in a haystack when the haystack is laid out for it. It does not tell you whether the runtime can hold the workspace that a normal agent request builds.

The first real request killed the 1M engine. A normal agent request from Pi on a single user, over a workspace that needed 703 MiB of locked memory when only 514 MiB was free. I lowered the router’s output allowance to try to protect it. It did not help. A second Pi request with a 19,118-token prompt and only 5,120 requested output tokens hit the same fatal allocation and took the engine down.

That second failure is the whole lesson in one request. Limiting output did not protect the 1M profile, because the problem was never the answer length. It was the prefill and workspace shape the prompt built before a single token came back. A benchmark that only varies retrieval depth will never see it. A real client will find it in one run.

The 650K profile answered the same way it would be used. It passed through Pi on two hosts and through OpenClaw, with high reasoning selected and no silent model fallback. A live 50,000-token request hit the model-specific 32,768 cap and returned an explicit clamp warning instead of a silent mutation. That is the operating contract that survived: 650K is repeatable for a single coding user, 1M remains visible as failed client-facing evidence, and every other GPU workload is mechanically excluded while the pair runs exclusive.

Why Promotion Is a Guarded Transaction, Not a Config Edit

The mistake is not that I tried 1M. The mistake would have been trusting the synthetic evidence enough to leave it. So in anvil-serving, promotion is the one operation that changes what callers actually get, and it is treated that way.

A benchmark never promotes a model. A recorded result can recommend a change. A person authorizes it. That separation is why the evidence layer can be trusted: nothing in the measurement path can quietly alter what is serving. serves promote stages a candidate recipe, qualifies it, swaps the role, and keeps a rollback path open the whole time. Preview before you commit, then confirm. If the candidate fails its gate, the transaction restores the previous state instead of leaving a half-promoted role.

That transaction is not a formality. It is the difference between a system that reports what it wants to be true and one that can only change what callers reach after someone confirms it.

Let me make the gate visible, because the abstract sounds stronger than it is until you see which rows a model has to clear.

Which gates does each profile clear?

Toggle the gate. Synthetic capacity passed the 1M profile; real client traffic did not. This is the row a spreadsheet cannot see.

Synthetic gates Real client traffic

Measured evidence (2026-08-02 promotion session)

Profile	Needle retrieval	Synthetic / protocol	Real client	Decision
128K	baseline	stable	stable	kept baseline
650K	~640K	passed	survived real Pi / OpenClaw	promoted as llm.primary
1M	~985K	passed	crashed twice	rejected, evidence kept

The Same Discipline Broke Down in the Release

The model gate was not the only place this lesson bit in the same session. The release that shipped the 650K promotion, version 0.21.0, passed CI and a merged PR, and then the deployed controller rejected a valid promoted manifest because the hard container did not mount the router profiles the new transactional activation depended on.

Clean CI and a merged PR do not prove that the live deployment has all its files. Transactional router activation added target and rollback profile files to the controller’s runtime contract, but the hardened Docker Compose deployment did not mount them. The missing files caused an outage after the release shipped. A hotfix added the read-only mounts and a regression test that derives every required profile from the manifest, so a future release cannot forget a dependency the same way.

The principle is identical to the model decision. Something validated in one shape, the test suite, does not automatically hold in the shape it is actually used, the deployed container. In both cases the fix was to make the real consumption path part of the gate: for the model, a client-shaped smoke; for the release, a manifest-derived dependency check.

Evidence Is a Floor, Not a Ceiling

I want to be honest about the narrowness of what worked. The 650K win ran with about 94 MiB of reserve headroom after the configured reserve. That is not a margin I would bet on a shared or co-resident box, and I do not treat it as a promise about anyone else’s hardware. It is a number I would measure again on a different host before I generalized it.

That is the tension the whole post turns on. The imported instinct is that more is better, that the 1M result, near-limit retrieval on better hardware, is the obvious winner. Discipline is not the enemy of conviction here. It is what lets a real conviction survive contact with reality. I did not want 650K because 1M was scary. I wanted the profile that real traffic could not break, because a local substrate only buys you the posture if it actually stays up.

A benchmark tells you what a model can do when everything is arranged for it. Real traffic tells you what it does when the work is real. There is no model so good that it trains that difference into you for free. You only learn it by letting production decide, and by keeping the human hand on the switch between what a test suggests and what people actually get. That hand is the whole platform. The anvil stays. The hammers rotate.