Qwen3.6-27B KV Cache Quantization in vLLM: Accuracy, Memory, and Speed
A smaller KV cache enables longer sequences and higher concurrency with virtually no loss in accuracy.
Long-context inference is often limited less by the model’s advertised context window than by the memory and bandwidth required for its KV cache.
During generation, attention layers store keys and values for every previous token. As context length and concurrency increase, this cache can consume tens of gigabytes and become a major memory-bandwidth bottleneck.
KV-cache quantization reduces that cost by storing keys and values at lower precision. Moving from 16-bit to approximately 4-bit storage can shrink the cache by nearly four times, enabling longer contexts and more concurrent requests.
In practice, however, a method that works well in a research implementation may perform poorly inside an optimized engine such as vLLM or llama.cpp. Quantization can add computational overhead, require specialized kernels, reduce hardware compatibility, and interfere with features such as speculative decoding.
In this article, I compare three 4-bit KV-cache formats in vLLM, turboquant_4bit_nc, int4_per_token_head, and nvfp4, using Qwen3.6-27B. I evaluate their memory usage, accuracy, token generation, and inference speed.
Acknowledgments
Verda provided the B200s and RTX Pro 6000s to run the experiments described in this article.
Verda is the full-stack frontier AI cloud, built for high-performance inference, training, and agentic workloads with data privacy and sustainability at its core.
You can check them out here. There is a $50 coupon that you can redeem in your Verda account, after provisionning it with $5, to try their GPUs.
Coupon code:
KAITCHUP-50Follow these instructions to redeem it.
Qwen3.6-27B KV Cache Size
I’m going to compare KV-cache memory usage across different quantization methods. First, let’s establish the memory required for Qwen3.6-27B’s native 262,144-token context window.
Qwen3.6-27B has 64 language layers, but only 16 are Gated Attention layers that maintain a full KV cache. The remaining layers are Gated DeltaNet blocks, so they are not included when calculating the conventional attention KV cache.
Each Gated Attention layer has four KV heads, with a head dimension of 256.
In BF16, the KV cache for these full-attention layers consumes approximately 17.2 GB. This is substantially less than for models that use full attention in every layer, but it is still significant relative to the model weights themselves, which occupy roughly 54 GB.
On a device with 96 GB of memory, that leaves:
96 − 54 − 17.2 = 24.8 GB
This is not enough to run several full-context requests in parallel, for example, multiple sub-agents, especially once additional overhead is included. CUDA graphs, temporary buffers, runtime allocations, and other inference-engine components all consume extra memory.