A Billion Tokens to Ship One Feature. Forty-Six Million Were Real.

The views expressed here are my own and do not represent those of any current or former employer.

I let a fleet of agents run for two days to ship one release, and when I added up the token meter afterward it read a little over a billion.

A billion tokens to ship one feature sounds like the setup for a lecture about waste. It is not. The interesting part is the opposite: almost none of that billion was the model doing work. When I split the meter into what was actually generated versus what was merely re-read, the generated slice was about five percent of the total. The other ninety-five percent was the same context, read again, and again, every turn. That gap is not an accounting curiosity. It is the difference between a number that scares you and a number that tells you whether your own hardware can carry the load.

This is the post where I stop arguing about the local box from a whiteboard and start reading the meter.

The run, and the wall it hit

The feature was a multi-PRD rework of Anvil, my state ledger, and it shipped as v0.3.0. I ran it the way I have been running everything lately: I set a goal and success criteria, said keep going until done and report in the morning, and let the loop self-verify against CI and a full suite, escalating only the calls a machine should not make alone. Across two sessions it ran about forty-four hours, drove roughly nine hundred background subagents, and merged twenty-one pull requests into a release.

It also hit the wall I keep saying is real, and this time I have it in the log instead of in my memory. Partway through, the provider started returning 529 overloads, the “server is overloaded, try again” error, and they killed review subagents mid-run and forced retries that ate wall-time. The retro filed one detail under “where we got lucky”: the overloads happened to strike review work, which is retryable, and never a merge or a release, which is not. That is the capacity problem stated precisely. It is not that I cannot afford the next token. It is that the provider, under my load, sometimes will not hand it over, and I do not get to choose which request it drops.

I am not the only one who saw it. The provider’s own status page logs more than twenty Claude service incidents this month alone, including a multi-model error-rate event on the same morning my build run was stalling. The 529s in my log were not a local fluke. They were the visible edge of a provider-wide bad day.

Let me be precise about the error, because the precision matters. A 529 is an availability failure, the provider overloaded. It is not the same as exhausting my flat-rate quota, which is a different ceiling I have not separately measured yet. But both are the same shape of problem: a bucket I do not control, emptying on a clock I do not set, while real work waits.

A billion tokens, and almost none of it was writing

Here is the anatomy of that meter, from one of the two sessions, which processed about 787 million tokens on its own.

cache read          773,588,663   the context re-read every turn
cache creation       10,290,870   writing new context into the cache
main output           2,770,177   tokens the orchestrator actually generated
fresh input             233,509   genuinely new prompt text
total processed     786,883,219

Look at the top line. Three-quarters of a billion tokens, in a single session, is cache read: the system prompt, the skills, the tool schemas, the task packet, all of it re-read on every single turn because that is how a stateless model stays oriented. It is real traffic and it is cheap per token, but it is not the model writing anything. It is the model re-reading the room before it speaks.

The tokens that represent actual generated work, across both sessions combined, came to about forty-six million. Against a billion processed, generation was roughly five percent.

This is the number people get wrong when they price a local box, and I almost got it wrong too. If you take the billion-token meter and divide it by a local generation rate, you conclude your hardware would need weeks and you give up. But you would be dividing a prefill number by a decode rate, and those are different machines doing different work. Prefill, reading that giant repeated context, is exactly what a local serving stack with prefix caching is built to make nearly free: the repeated prefix is a cache hit, not a re-computation. The only number that maps to your box’s real cost is the generated slice. Size the box against the five percent, not the hundred.

The number that actually sizes the box

So the question becomes answerable. Not “can my box process a billion tokens,” which is the wrong question, but “how long would my box take to generate the forty-six million tokens that were real work.”

My one honest local measurement is about two hundred tokens per second, from a coder model on the 5090 during an earlier eval. The 96GB card is newer and I have not benchmarked it under this load yet, so everything faster than that two hundred is a target I have to earn, not a result I can claim.

Local generation rate	Hours to emit ~46 million tokens	Status
200 tokens/sec (measured, 5090)	about 65	real
400 tokens/sec	about 32	target
800 tokens/sec (96GB, batched)	about 16	target
1500 tokens/sec (aggressive aggregate)	about 9	target

Read the first row, because it is the only one I have earned. At my measured rate, the box would need about sixty-five hours of steady generation to emit what the cloud emitted in about fifty-two hours of wall-clock. Same order of magnitude. And that measured rate is a single stream; the cloud run got its speed from running nine hundred agents at once, and the thing a local serving stack does well is batch concurrent requests, so the 96GB card’s real number under fan-out is plausibly well above two hundred. I just have to prove it, not assume it.

The headline is quieter than “local replaces the cloud,” and truer: on raw generation capacity, my own box is not the bottleneck. It can plausibly emit the work in a comparable stretch of time. The thing that was supposed to be the hard constraint, throughput, is not the hard constraint.

Capacity is not the same as competence

Here is where I have to stop the good news before it oversells, because the meter only measured one thing and it is not the thing that matters most.

Forty-six million generated tokens from a frontier model is not the same as forty-six million from a local one. They are the same quantity and they are not the same quality. The cloud run produced twenty-one merged pull requests and a clean release; a local model emitting the same token count produces the same volume of text and tells you nothing about whether that text would have passed review. The capacity math says the engine is big enough. It says nothing about whether the work is correct.

That second number, the rate at which local work is actually right versus quietly wrong, is the one I have not measured. It is the whole reason I keep the local tier’s output in a needs-review state instead of auto-merging it. Throughput I can read off a meter in an afternoon. Correctness I have to earn the slow way, on my real backlog, counting how often a tight task packet lets a cheap model land a change that survives the same gate the cloud work has to clear. The meter answered the easy half. The hard half is still open, and pretending otherwise would just be a prettier version of the mistake I started this post warning about.

The lesson the meter taught

The token bill lies about where the work is. It quotes you a number dominated by the cheapest thing the system does, re-reading its own context, and buries the one line that actually costs your hardware something. If you are deciding whether to put real load on a machine you own, the move is not to flinch at the big number. It is to split the meter, find the slice that is genuinely generation, and size your box against that, in the unit your box is actually billed in.

I went looking for a reason to feel good about an expensive purchase and found something more useful instead: a measurement that settles one question and sharpens the next. The box can keep up. Whether it can be trusted is a different number, and I am going to go take it.