I Didn't Need a Bigger Computer. I Needed Room to Think.

The views expressed here are my own and do not represent those of any current or former employer.

The first time my local agents felt like coworkers instead of a demo, they ran out of room before they finished the job.

I had three of them going on a single RTX 5090, served through SGLang, pulling real weight on real tasks. It worked. And then, every single time, compaction hit like clockwork: the agent would get rolling, fill its window, compact, lose the thread, and I would be back to babysitting. Re-feeding context. Re-explaining. Re-pointing at the file it had just been looking at.

The reflex is to assume the model is too small or too dumb. It was neither. The bottleneck was a number, and the number was usable context.

The arithmetic that ruined it

The bottleneck had an address. I was serving a 35B model (Qwen3.6-35B-A3B) through SGLang, capped at a 32,768-token window, and the agent loop died on turn one, every time, before it did real work. Single-shot prompts were fine. The multi-turn loop was not. The cause was one subtraction.

The runtime decides when to compact a conversation with a single rule:

compact when:  contextTokens > contextWindow - reserveTokens

reserveTokens is headroom carved out of the token budget, not out of VRAM, and the trap is its floor. OpenClaw enforces a reserveTokensFloor of 20,000 for embedded runs. That is a rounding error on a 200K cloud window and a catastrophe on a small one:

contextWindow       = 32,768
reserveTokensFloor  = 20,000
threshold           = 32,768 - 20,000 = 12,768 usable tokens

The agent’s own fixed overhead on turn one, its system prompt, the workspace files it injects, and the coding tool schemas, already exceeds that 12,768. So compaction fires before the first real token, a second compaction is blocked by an “already compacted” guard, and that block surfaces as a hard error instead of a no-op. The run is dead on arrival.

The fix was two lines:

reserveTokensFloor  = 0
reserveTokens       = 8,192
threshold           = 32,768 - 8,192 = 24,576 usable tokens

Reserve	Threshold (window - reserve)	Usable	Reserved	Turn-1 result
20,000 (default floor)	32,768 - 20,000 = 12,768	39%	61%	deadlock
8,192 (the fix)	32,768 - 8,192 = 24,576	75%	25%	runs

A flat constant auto-scales, which is why it works fleet-wide: 8,192 is a 25% bite out of a 32K local window and about 4% of a 200K cloud window, so one global setting serves both tiers. I chose 8,192 because it equals the model’s max output. The old 20,000 had been tuned for frontier windows, where nobody feels it.

Two honest notes. The 32,768 is a cap I set for local serving, not the model’s native window, which is far larger. And 24,576 is the compaction threshold, the line where it triggers, not free space: the turn-one overhead still lives inside it. The point is not the exact ceiling. It is that on a small window the reserve, not the model, was eating the room.

So the agents were fast and they were capable, but they were claustrophobic. Not dumb. Cramped. There is a difference, and the difference is the whole story, because cramped is a hardware problem and dumb is not.

That reframed the goal. I did not need a smarter model or even, exactly, a bigger one. I needed more usable context per agent while several agents ran at once. That is a different shopping list.

The model you run is the other half of the math

The card sets the ceiling; the model you load decides how fast you hit it. KV cache, the per-token memory each agent needs to hold its context, swings wildly with the model, and not in the way you would guess. Here is the cost of a single 100,000-token agent at FP8:

Model	Architecture	KV per 100K-token agent (FP8)
Nemotron-3 Nano 30B-A3B (NVIDIA)	Mamba-2 + MoE hybrid	~0.3 GB
Nemotron-3 Super 120B-A12B (NVIDIA)	Mamba-2 + MoE hybrid	~0.4 GB
Gemma 4 26B-A4B (Google)	sparse MoE, sliding-window	~1.0 GB
GLM-4.7-Flash 30B-A3B (Z.ai)	MLA + MoE	~2.7 GB
Gemma 4 31B (Google)	dense, sliding-window	~4.1 GB
Qwen3-Coder 30B-A3B (Qwen)	sparse MoE, full attention	~4.9 GB

A 120B model (Nemotron-3 Super) holds less KV than a 30B one (Qwen3-Coder), and not by a little. Parameter count is not what drives the number; attention design is. A Mamba hybrid keeps a fixed recurrent state on most layers, a sliding-window model bounds the cache to its window, MLA compresses K and V into one small latent, and a plain full-attention model pays full freight on every layer. So the card you can afford and the attention architecture you pick decide how many agents fit, long before parameter count does. (These figures assume an FP8 KV cache, one byte per element; at the more common FP16 they double. The two sliding-window rows, the Gemmas, are approximate, since how much the window saves grows with the context length.)

Three ways out, priced

There were three honest roads to more room, and they are not equal.

The first was a unified-memory box: a DGX Spark or a Strix Halo machine with 128GB of memory the GPU can address. It fits enormous models. But unified memory is a capacity tool, not a throughput tool. The bandwidth sits around 256 to 273 gigabytes per second, and for an interactive agent loop where you are waiting on tokens, that is slow enough to feel it on every turn. Plenty of room, not enough speed.

The second was the proper workstation: a Threadripper platform, registered ECC memory, and room for multiple large cards. This is the right answer if money is not part of the problem. It was. Between the platform, the memory, and the cards, it blew well past what this project could justify, and most of what I would be buying was headroom I had not yet proven I needed.

The third was the boring one. Keep almost everything I already owned and add exactly the one part that was actually scarce: VRAM with real bandwidth behind it.

I took the third road, and the discipline of it was the point. The question I asked at the shop was not “what is the best machine I could build,” it was “what can I keep.” I kept the Ryzen 9800X3D, the 96GB of DDR5, the existing NVMe, the existing cooler, and the 5090. I added a single RTX PRO 6000 Blackwell Max-Q, the 96GB card at around 300 watts and roughly 1.8 terabytes per second of bandwidth, plus the few parts required to host two big GPUs cleanly: a workstation motherboard with a sane lane layout, a larger case, a second NVMe, and a power supply with real headroom.

I did not even buy more RAM. Four-DIMM kits on this platform are a stability gamble, and 96GB of VRAM is the capacity that mattered for this workload, not another 96GB of system memory at a price that had gone briefly insane.

Why both cards live in one box

The clean answer for most people is to split: put the big card in a server and leave the gaming card in a desktop. I deliberately did not. The whole point of this machine is exploration, and a split removes the easiest path to the experiments I actually want to run.

I want to be precise about what the second card is and is not. Ninety-six gigabytes plus thirty-two is not a hundred-and-twenty-eight gigabyte pool. They are two separate pools, no NVLink between them, and tensor-parallelism across mismatched cards bottlenecks on the smaller one. So I am not pretending I built a 128GB GPU.

What I built is a primary long-context server (the 96GB card) and a second independent worker (the 5090) on the same node. The 5090 stays the display and gaming card, and it doubles as a second model endpoint for the cheaper, smaller roles: the reviewer-screen, the file-enumerator, the embeddings. Same box means the two can share a workflow without a network hop, and it leaves a clean path to test pipeline-parallel splitting later if I ever want a model that spills past 96GB. Part of the value here is honestly the joy of the build. That joy aligns with the architecture instead of fighting it.

The trick that turns 96GB into room for five agents

More VRAM alone would have helped. But the thing that actually fixes the original problem is not the raw gigabytes, it is a serving feature.

My agents share a large common prefix: the same system prompt, the same skills, the same canonical project state, on the order of nine thousand-plus tokens, loaded identically by every agent. A serving stack with prefix caching (SGLang’s RadixAttention, or vLLM’s automatic prefix caching) stores that shared prefix’s KV cache once instead of once per agent. Five agents sharing a 20K prefix with a 5K unique tail each do not cost five times 25K of cache. They cost the 20K prefix once plus five times 5K, which is well under half. That is roughly a 3x multiplier on how many agents fit in the same memory, and it grows as the shared fraction grows.

On the 5090 I never had the headroom for that multiplier to matter. On the 96GB card it is the whole game. The architectural cost is one rule: the shared prefix has to be byte-identical across agents, so no per-agent timestamps or interpolation inside the cached region. One stray character and the cache match breaks. It is the same optimization the cloud charges you a tenth of the price for when it bills cached input.

What I get to do with the room

The reason any of this matters is the work pattern. The way I run Anvil, my durable system of record for agent teams, is not a single model call. It is three to five agents working in waves: bounded, well-specified tasks pulled from a queue, executed by whichever runtime fits, with their evidence written back to a record that survives the chat window. The bottleneck for that pattern was never the size of any one model. It was KV cache and context budget under concurrency. The exact thing the 96GB card and prefix caching address.

So the target is modest and concrete: three to five local agents with enough room each to actually finish a task without compacting mid-thought, doing the bounded, low-risk middle of the backlog, while the rare high-stakes reasoning stays on a frontier model in the cloud. Local for volume. Cloud for stakes. That is the architecture I am building toward, not the default I run today, and the local work never leaves the machine, which is its own quiet payoff.

The honest part

I am writing this with the machine still on the bench. It is being assembled, and I do not have it back yet, so every number above the line “what I want it to do” is a target, not a measurement. The promise of three-to-five roomy agents is a hypothesis I am about to test, not a result I am reporting. When I have it, I will run my real backlog on it for a couple of weeks and measure the things I have only modeled: how much usable context I actually get per agent, how many run before throughput falls apart, and whether the cheap local model’s output is good enough often enough to be worth the electricity. If the data disagrees with the whiteboard, the whiteboard loses.

What I do have is one early result from the 5090, the card I already own. On a small suite of six deterministic coding tasks, the local model matched a cloud frontier model on correctness, six for six, and came back faster on wall-clock. It is a single run on six small tasks, so it proves the plumbing works, not that local equals frontier on the hard problems. But it is the first real sign that the cramped model, once it had room to think, actually does the work.

But the decision underneath it I am confident in, because it is not really about GPUs. The capacity problem that sent me shopping in the first place, running dry on both flat-rate plans at once, is its own story. The trap underneath it was thinking I needed a bigger computer. I needed one scarce resource, usable context under concurrency, and the cheapest way to buy it was to keep the machine I had and add the single part that was actually missing. Build for the constraint you measured, not the rig you imagined. The orchestration churns, the models get swapped, the next card comes out in a year. The thing worth owning is the smallest box that gives the work room to happen.