Testing Qwen3.8-27B KV cache dtypes

TL;DR

FP8 approximately doubles the available KV-cache capacity compared to FP16 without showing a meaningful quality loss in these tests. FP8 does not substantially improve ordinary single-request generation speed compared to FP16. Extending the context window with YaRN up to 1M works very well even in the archival tests, but a cold request near 959k took 41 minutes to complete.

Between the four 3090s in my mlrig I have 96 GiB of VRAM. The currently latest Qwen3.8-27B with FP16 weights fits more than nicely in this setup as Qwen3.8-27B has a 51.75 GiB FP16 checkpoint and occupies approximately 52.04 GiB across the four GPUs when loaded by vLLM. With gpu-memory-utilization=0.95, tensor parallelism 4, and a native 262,144-token context, we can get the following:

Specification	FP16 KV cache	FP8 KV cache
KV datatype	`float16`	`fp8` E4M3
Bytes per cached value	2	1
KV memory per GPU	8.63 GiB	8.23 GiB
KV memory across four GPUs	34.52 GiB	32.92 GiB
Engine token capacity	552,731	1,036,523
262,144-token concurrency	2.11x	3.95x
CUDA graph estimate per GPU	0.14 GiB	0.51 GiB
Cache block size	16 tokens	16 tokens
Automatic prefix caching	Enabled	Enabled
Dynamic KV-scale calculation	Disabled	Disabled
FP8 scaling	N/A	`1.0` fallback; dynamic calibration disabled for hybrid model relevant issue

FP8 is tempting as the KV cache is nearly doubled, giving a lot more room for concurrency and possibly a YaRN-extended context. Not entirely doubled because FP8 requires a different attention backend resulting in a slightly larger CUDA graph compared to FP16.

This led me to want to test if there is actually a difference between FP8 and FP16 for the KV cache of Qwen3.8-27B – they say that the quality of particularly long-context conversations can degrade.

Test design

I first compared FP16 and FP8 at Qwen3.8’s native 262,144-token context. The model weights remained FP16 in every run; only the attention KV-cache datatype changed. Both configurations used the same model revision, four GPUs with tensor parallelism, two scheduler sequence slots, an 8,192-token batch limit and the same 95% GPU-memory allocation.

The benchmark contained six groups of 20 deterministic questions:

arithmetic with signs, precedence, large products, division, remainders, exponents, decimals and fractions;
Unix permissions, including octal and symbolic forms and ordered chmod operations;
IPv4 subnet calculations from /8 through /32, with explicit conventions for /31 and /32;
backup-retention sets with disjoint, nested and overlapping daily, weekly and monthly snapshots;
Python mutation, rebinding, shallow and deep copying, repeated nested objects and mutable default arguments;
stable sorting of between six and 25 people, including ties, descending order and compound keys.

These 120 questions ran without thinking and at temperature zero (aka. greedy decoding). Answers had to match a specified JSON structure exactly, so a correct explanation in the wrong format still failed. The final two questions from each group were also repeated with xhigh reasoning and Qwen’s recommended thinking parameters, giving 12 separate reasoning tests with a 32,768-token output allowance.

I also gave the model six Docker Compose design tasks. They ranged from a small static site to multi-service application and database deployments with health checks, restricted networks, persistent data, secrets, resource limits and log rotation. The generated YAML was parsed independently, and semantic checks tested whether the requested relationships and restrictions were actually present.

The complete task catalog contains every prompt and expected answer used in the comparison.

Native-context result

Native cache	Greedy diagnostics	Xhigh repeats	Compose designs	Archive tasks
FP16	82/120	12/12	6/6	15/15
FP8	80/120	12/12	6/6	15/15

The two extra FP8 misses were one Unix-permission answer and one stable-sort answer. The model produced the correct rwx form but the wrong octal value in the former, and changed the order of two tied entries in the latter. A two-question difference amounts to a 1.67% difference and is not enough to establish a general quality loss.

Generation speed was effectively tied. Median decode throughput was 26.54 tokens/s with FP16 and 26.52 tokens/s with FP8 for the no-thinking questions. The xhigh medians were 24.59 and 24.61 tokens/s respectively. FP8 therefore provided much more cache capacity, but it did not make ordinary single-request generation faster in a meaningful way.

In an earlier concurrency test I submitted two independent 225,065-token requests at the same time. The complete pair took an average of 523.55 seconds with FP16 and 493.78 seconds with FP8. Peak sampled cache usage was approximately 81% and 43% respectively, matching the fact that the KV cache is approximately doubled from FP16 to FP8. FP8 was approximately 5.7% faster in this particular cache-intensive concurrent workload, but ordinary token generation showed no meaningful speed difference. The result includes the effects of lower cache pressure and a different attention backend, so it cannot be attributed solely to FP8 arithmetic.

Long-archive tests

Simple questions do not exercise a large KV cache, so I generated synthetic archives containing thousands of similar but distinct fictional operations records. Sparse authoritative facts were inserted at known positions, including near the beginning, middle and end of the context. A separate cache salt made every measured request cold, preventing a previous prompt from supplying a prefix-cache hit (and I explicitly checked that there were 0 cache hits).

The archive suite tested five different operations:

Retrieve eight exact codes placed at different depths.
Select the highest revision of records that had been updated repeatedly.
Join identifiers from one part of the archive to invoice values elsewhere.
Follow chains of aliases spread across the document.
Aggregate ledger entries into exact totals.

This is more demanding than finding one obvious needle. The join, chain and aggregation tasks require the model to retain several separated facts and use them together. Insertions were distributed from roughly 1% through 99% of the generated filler, and answers were graded both as complete JSON objects and as individual recovered fields.

At native context, both FP16 and FP8 passed all 15 archive tasks and recovered all 63 requested fields at approximately 7k, 64k and 224k actual input tokens. The earlier five-code archive series also produced 140/140 correct fields across both cache types and repeated runs. I therefore found no long-retrieval quality loss from FP8 within the native context tested here.

Extending the context with YaRN

Qwen3.8-27B has a native context of 262,144 tokens and documents extension up to one million tokens using static YaRN scaling. I tested two FP8 configurations in addition to the native comparison:

YaRN factor 2 with a 524,288-token context;
YaRN factor 4 with a 1,000,000-token context.

The factor-2 server reported capacity for 1,040,888 cached tokens, approximately 1.99 complete 524,288-token sequences. It completed every request without a transport error or truncation. The short diagnostics scored 81/120, all 12 xhigh repeats passed, and five of the six Compose designs passed. The failed Compose answer used hostIP where the Compose schema requires host_ip.

All native-sized archive tests still passed under factor 2. At approximately 448,800 input tokens, four of five responses passed the strict JSON score. The remaining answer recovered all 21 requested fields but added explanatory prose around the JSON, making this a formatting failure rather than lost archive information. Each cold request at this length took roughly 11.5 minutes.

The factor-4 server accepted the full one-million-token setting and processed actual inputs as large as 958,805 tokens while retaining a 32,768-token output allowance. Across 35 archive tasks from about 7K to 959k input tokens, 33 passed the strict whole-answer check. All five tasks passed through approximately 699k.

At approximately 899k, one ledger aggregation was wrong by two. At approximately 959k, one join recovered and calculated every requested value correctly but again added prose around the required JSON. Counting the correct embedded JSON, 146 of 147 requested archive fields were recovered over the entire extended suite. The five largest cold requests averaged 2,461 seconds, or 41 minutes, each. Long context worked, but initial processing time became the dominant cost.

Configuration	Greedy diagnostics	Xhigh repeats	Compose designs	Largest tested input
FP16, native context	82/120	12/12	6/6	approximately 224k
FP8, native context	80/120	12/12	6/6	approximately 224k
FP8, YaRN factor 2	81/120	12/12	5/6	approximately 449k
FP8, YaRN factor 4	79/120	11/12	6/6	approximately 959k

Static YaRN changes the positional encoding for every request, including short ones. The small and mixed score changes above therefore combine the effects of FP8 cache storage and YaRN scaling.

Conclusion

Within Qwen3.8’s native context, these tests found no meaningful performance difference and no consistent quality winner between FP16 and FP8 KV cache. FP8 recovered all long-archive facts tested at native length and nearly doubled the available token capacity. That makes it the practical choice when the intended context or concurrency does not fit with FP16.

The extended tests also show that accepting a one-million-token is taxing. Cold request near 959k took about 41 minutes which limits the use-cases, however, the test was designed to circumvent the prefix cache and in a normal chat situation it would likely be faster (I’m getting consistently 96-97% cache hits under normal OpenCode chat circumstances). Retrieval went surprisingly well I must say, though there was one numerical error near 900k.

For my normal deployment I settled on FP8 KV cache with a 524,288-token context and YaRN factor 2. It gives enough room for approximately two maximum-length sequences and it does not impose the extreme cold-prefill cost of the one-million-token configuration.