KV Cache Hierarchy: How Modern LLM Serving Actually Uses Your VRAM
aiengineering
A deep dive into paged attention, continuous batching, and the GPU-to-NVMe KV cache hierarchy that decides whether your LLM serving stack scales or stalls.
Decode is memory-bound, not compute-bound. Every token an LLM emits requires reading the entire KV cache from VRAM — which means the shape of your cache, not the size of your model, sets your throughput ceiling. This deep dive unpacks the three mechanics that modern serving stacks (vLLM, TensorRT-LLM, SGLang) use to squeeze more requests through the same GPU.
Paged attention: fixed-size blocks, zero fragmentation
Traditional attention allocates a contiguous KV buffer per request sized for the worst-case sequence length. Short prompts waste most of it; long prompts can't fit. Paged attention breaks the KV cache into fixed-size pages (typically 16 tokens) and maintains a block table per request, borrowing the virtual memory idea from operating systems. Fragmentation drops from 60–80% to under 4%, and prefix pages can be shared across requests that start with the same system prompt.
Continuous batching: interleave, don't drain
Static batching waits for the slowest request in a batch to finish before starting the next batch — a disaster when sequence lengths vary by 10×. Continuous batching admits new requests into the batch at every decode step, evicts completed ones immediately, and keeps the GPU saturated. Real-world throughput gains are typically 2–4× at the same latency.
The three-tier cache hierarchy
| Tier | Medium | Latency | Use case |
|---|---|---|---|
| Hot | GPU VRAM | ~1 µs | Active decode sessions |
| Warm | CPU RAM | ~100 µs | Paused sessions, hot prefixes |
| Cold | NVMe SSD | ~10 ms | Long-tail conversation history |
Offloading policy matters more than tier count. Evict LRU pages first, but pin shared prefixes (system prompts, few-shot examples) in VRAM — they amortize across every request that uses them.
The tenant-isolation trap
Prefix sharing is a throughput win and a security risk. If two tenants' requests share the same system prompt, their KV pages can be reused — but if your cache key doesn't include a tenant ID, one tenant's cached context can bleed into another's response. This is the same class of failure that makes semantic caching dangerous in multi-tenant deployments.
If the serving layer is not where you want to spend engineering cycles, the Silverberry AI Platform handles tenant-isolated serving, caching, and governance for enterprise workloads.