The Model Was the Same. The Serving Configuration Was Not.

The views expressed here are my own and do not represent those of any current or former employer.

I had two Qwen3.8 27B serves running at the same time. One server sat on each of my two RTX PRO 6000 cards. Same 96 GB of VRAM per lane. Same runtime. Same native 262,144-token context. Same benchmark prompts.

Then the answers started arriving at very different speeds.

The BF16 checkpoint decoded at 26.9 tokens per second. The official FP8 checkpoint reached 47.9. When I enabled the multi-token prediction head already inside the FP8 checkpoint, decode reached 94.8. The model name had not changed. The experience had.

That was the moment the test stopped being a simple quantization comparison. I was not benchmarking one model against another. I was measuring how four serving choices changed the personality of the same model.

The Same Architecture Arrived in Two Different Boxes

Qwen’s official BF16 revision is a 27-billion-parameter dense multimodal checkpoint. Its 64-layer backbone mixes 48 linear-attention layers with 16 full-attention layers, includes a vision tower, and carries a built-in multi-token-prediction draft head. The official FP8 revision is that same architecture with block-scaled FP8 weights.

The disk and memory difference was immediate. BF16 carried 51.7 GiB of weights and loaded at roughly 51.1 GiB. Official FP8 carried 28.7 GiB and loaded at roughly 27.64 GiB. I ran BF16 as the full multimodal configuration and FP8 as text-only, which skipped the vision encoder and left more room for KV cache.

The pinned vLLM recipe was the upstream prior. It was not the result. That recipe had been verified on GB300 hardware and included a four-GPU FP8 example. My job was to translate it to two separate SM120 workstation cards, run one TP=1 serve on each, and then prove what survived the translation.

The Decode Result Was Real, but It Was Not Prefill

The short workload had a median rendered prompt of 3,613 tokens and produced roughly 45 output tokens per request. That distinction matters. Time to first token measures scheduling plus prompt processing. Decode measures what happens after the first token. End-to-end latency includes both.

Official FP8 improved every practical part of the single-request text lane. Median first-token latency fell from 848 to 793 milliseconds. Effective prefill rose from 4,244 to 4,553 tokens per second. Decode climbed from 26.9 to 47.9 tokens per second, while end-to-end latency fell from 2.456 to 1.694 seconds.

Then MTP changed the shape. It did almost nothing for first-token latency, because speculative decoding cannot help before generation starts. After that first token, it nearly doubled decode to 94.8 tokens per second and cut inter-token latency to 10.5 milliseconds. The answer felt different because the back half of the request was different.

At five concurrent requests, the story narrowed. Baseline FP8 produced 51 aggregate output tokens per second. MTP reached 54. That is useful, but it is not another doubling. Speculation bought interactive latency much more than saturated throughput.

This is the part benchmark headlines flatten. A configuration can be the clear winner for one person waiting on an answer and barely move the server’s maximum batch output. “Faster” is not one number. It is a workload declaration.

Long Context Did Not Erase the FP8 Advantage

Qwen3.8’s hybrid attention structure is interesting here. Only one in four layers runs full attention; the other 48 maintain a linear-attention recurrent state. I did not isolate that architectural choice, so I will not credit it for the result. I can only say the measured curve remained usable and FP8 stayed ahead as the prompt grew.

At 31,225 prompt tokens, FP8 returned the first token in 5.88 seconds against BF16’s 8.05. At 139,428 tokens, the comparison was 42.04 against 53.44 seconds. At 241,250 tokens, FP8 reached the first token in 104.71 seconds, while BF16 took 125.14.

Every row passed the hidden retrieval check. That does not make this a broad long-context reasoning score. It proves something narrower and more useful: both configurations could retrieve the target at those actual prompt sizes, and the FP8 serving advantage remained visible near the top of the native window.

Prefix Reuse Changed the Workload, Not Just the Score

The largest jump did not come from quantization or speculation. It came from not doing the same work twice.

I sent five simultaneous requests that shared a 30,000-token prefix. Think of an agent session carrying the same system prompt, tool definitions, skills, and project state into every turn. With prefix caching disabled, the repeated burst still took 16.39 seconds to reach the median first token and produced 9 aggregate output tokens per second.

With caching enabled, the cold pass reached the first token in 9.179 seconds. The identical warm repeat reached it in 0.407 seconds and produced 142 aggregate output tokens per second.

That 73,934-token-per-second “effective prefill” number is not a magic GPU kernel. It is mostly work avoided. The engine reused the shared prefix instead of rebuilding it. The endpoint did not expose a cached-token counter, so I am calling this timing-based reuse evidence, not counter-backed accounting.

The distinction is important. If your workload is a stream of unrelated prompts, this result is mostly irrelevant. If your workload is a coding agent carrying tens of thousands of stable tokens from turn to turn, prefix identity becomes part of serving architecture. Cache invalidation is not a footnote. It is the performance model.

Memory Was the Other Half of the Benchmark

Speed was only one budget. The official FP8 text baseline reported room for 1,825,809 KV tokens, about 6.96 complete native-context windows. Switching only the KV cache to the model’s BF16 dtype cut that to 929,913 tokens, about 3.55 full windows.

Short-context performance barely moved. FP8 weights with BF16 KV decoded at 47.8 tokens per second, essentially identical to the 47.9-token FP8-cache baseline. The cost showed up in capacity, not latency.

That configuration still matters because vLLM warned that absent FP8 attention scaling factors defaulted to 1.0. The functional and repeated-quality gates passed, but that warning prevents me from calling FP8 KV quality-equivalent to unquantized KV. BF16 KV is the control I keep when I want to separate weight quantization from cache quantization. It is not faster. It is more conservative, and it costs roughly half the cache capacity.

I Tested the Files Before I Tested the Model

There was one checkpoint I deliberately did not run. The upstream recipe listed a third-party NVFP4 build. I did not have the same official-publisher provenance for it, so I stopped at the official Qwen artifacts.

For both official checkpoints, every local safetensors SHA-256 matched the pinned Hugging Face LFS identity. The snapshots contained no executable Python, pickle, PyTorch binary, native library, auto_map, or trust_remote_code path. Safetensors are not a complete safety guarantee. They are a much tighter loading boundary, and pinned hashes establish that the files I measured were the files Qwen published.

That is the order I want: identity, loader surface, then performance. A benchmark on an artifact I cannot confidently identify is just an expensive way to become uncertain faster.

What Passed, and What Still Has Not

Both official checkpoints passed JSON, streaming, batched tool calls, tool-result continuation, the Responses API, repeated session and tool checks, and adaptive reasoning at low, medium, and xhigh effort. The BF16 multimodal lane also passed 30 deterministic attempts across image, video, and mixed inputs.

I have not run SWE-bench on this candidate. I have not combined MTP with prefix caching. I have not completed the clean MTP batched-token tuning sweep that vLLM’s warning suggests. I have not tested TP=2, and I have not promoted either checkpoint behind a client-facing router alias. The current decision is challenger, no promotion.

What comes next is therefore specific:

1. Hold the FP8 baseline fixed and tune the MTP batch-token ceiling one variable at a time.
2. Run an independent matched quality A/B between FP8 KV and BF16 KV.
3. Combine the winning MTP setting with prefix caching and test both interactive and saturated workloads.
4. Put the candidate through the durable agentic, coding, long-context recovery, and real-client acceptance gates before any promotion discussion.

I started with two model repositories and expected a precision contest. I ended with a serving lesson. The checkpoint decides what capability enters the room. The configuration decides how much of that capability a real workload can reach, how quickly it reaches it, and how many other requests can fit beside it.

A model name is a label on the hammer. The benchmark has to tell you how it actually lands on your anvil.