Systems · Research analysis
Long context has a memory budget
A worked KV-cache calculation, and why FlashAttention, GQA, and PagedAttention solve different problems.
On this page
THE ESSENTIALS
- An advertised context limit is not a concurrency estimate.
- A hypothetical 32-layer GQA decoder uses 4 GiB of raw BF16 KV cache at 32,768 retained tokens.
- Attention kernels, cache allocation, and model architecture need separate measurements.
Our analysis
The capacity question
“How many users can one GPU serve?” has no single answer without a workload. A short request with a small generated answer and a long conversation with a large retained context make different demands. Concurrency, arrival rate, latency targets, and the distribution of prompt lengths all matter.
The first useful step is to write down a memory model. That will not predict throughput by itself, but it can reject an impossible plan and reveal which quantities need to be measured next.
Published evidence
Three mechanisms, three different contributions
FlashAttention changes how exact attention is computed, using tiling and attention to memory traffic. It should not be described as making every form of attention linear-time or deleting all inference cache costs. [1]
PagedAttention organizes the allocation and sharing of KV-cache blocks. It addresses waste and management overhead rather than changing the logical number of key and value elements required by a particular sequence. The vLLM design documentation explains a concrete implementation setting. [2, 4]
Grouped-query attention changes how query heads share key and value heads. It is an architectural design choice, not a server flag that can be applied to an arbitrary checkpoint with no modeling implications. The GQA paper also studies conversion through additional training. [3]
Analytical example
A calculation you can reproduce
For a conventional decoder that retains a key and value tensor at every layer, an idealized raw cache estimate is M = 2 × L × T × Hkv × D × b × B. L is the number of layers, T the retained token count, Hkv the key/value head count, D the head dimension, b bytes per stored element, and B the number of sequences. The factor of two accounts for keys and values.
Take a hypothetical model with 32 layers, 8 KV heads, head dimension 128, BF16 storage at 2 bytes per element, and one sequence with 32,768 retained tokens. The result is 4,294,967,296 bytes: exactly 4 GiB. At 8,192 tokens the same estimate is 1 GiB. Eight equal-length 32,768-token sequences would require 32 GiB of raw cache before model weights, workspaces, allocator overhead, and other state.
If an otherwise identical example used 32 KV heads, the raw cache would be four times larger. This is an arithmetic comparison of tensor shapes, not evidence that changing a trained model from 32 to 8 KV heads preserves its quality.
The estimate assumes full retained attention state in that representation. Sliding-window attention, different cache layouts, quantization, prefix sharing, and architectures with different state representations require a revised model. A nominal one-byte cache also has scale and metadata overhead; its accuracy tradeoffs cannot be inferred from the byte count alone.
EXPLORE THE CALCULATION
How context changes the raw cache
Fixed example: 32 layers and head dimension 128. Change the retained tokens, KV heads, and sequence count.
Excludes weights, workspaces, allocator and quantization metadata. This is tensor arithmetic, not a serving-capacity benchmark.Proposed experiment
From a byte count to a serving plan
Measure weight residency and non-cache memory under the actual serving configuration, then reserve headroom before allocating the remainder to cache. Use a distribution of retained lengths, not only a maximum or an average. Bursty arrivals and a few long requests can dominate the tail even when average utilization appears comfortable.
Benchmark prefill and decode separately. Prefill processes prompt tokens; decode incrementally generates new tokens while reading accumulated state. A configuration that improves one stage may not improve the other, and queueing connects both to the user’s observed latency.
Replay representative arrivals with short, medium, and long contexts. Record time to first token, inter-token latency, completion time, throughput, memory high-water mark, preemptions, and rejected requests. Include the latency distribution, not only an aggregate tokens-per-second figure. Report the scheduler and batching configuration with the model revision.
| Technique | Primary lever | A claim to avoid |
|---|---|---|
| FlashAttention | Attention computation and memory traffic | All long-context costs disappear |
| PagedAttention | KV allocation and sharing | Logical KV state becomes free |
| GQA | KV head count in the model architecture | Every checkpoint can switch without consequences |
| Shorter retained history | Amount of state carried forward | Answer quality is unaffected by truncation |
Our analysis
Quality is part of the budget
If a system shortens conversation history to fit more concurrent requests, test whether it loses the facts and constraints required for later answers. If it compresses a prefix or quantizes its cache, include difficult retrieval and generation cases. Higher concurrency is not useful if the resulting answers or interaction times fail the product’s requirements.
Our recommendation is to publish a capacity envelope: model and precision, prompt/output distributions, arrival pattern, hardware, concurrency range, and latency/quality limits. A single “users per GPU” figure leaves out the conditions that make it meaningful.
Scope & limits
What the numbers mean
The cache figures are exact arithmetic under stated assumptions and are reproduced by the downloadable program. They are not GPU measurements. No serving throughput, production concurrency, or model-quality result was measured in this analysis.
Reproduce the analytical examples
The standard-library Python program checks the probability, cache-memory and interval calculations, plus the small policy examples. It does not train a model or run the cited benchmarks.
A CLOSER LOOK
Ask this article.
Ask a question to find passages and their references. Answers are extracted from this article; no live AI model is connected.
READ THE ORIGINAL WORK
Sources & references
Sources reviewed 20 October 2026. Papers describe their own models and experiments; our proposed experiments are identified separately.
- 1FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness ↗
Dao et al. · 2022
arxiv.org - 2Efficient Memory Management for Large Language Model Serving with PagedAttention ↗
Kwon et al. · 2023
arxiv.org - 3GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints ↗
Ainslie et al. · 2023
arxiv.org - 4Paged Attention implementation notes ↗
vLLM maintainers · Living documentation
docs.vllm.ai
Keep exploring
All stories ↗
TrainingGRPO efficiency starts before the kernel
A practical investigation of sparse rewards, loss normalization, and what a faster training step actually proves.
RetrievalRAG should earn the right to answer
A claim-level design for retrieval, contradiction handling, citations, and useful abstention.
AgentsA coding agent is a system, not a score
How to compare coding agents without confusing model quality, scaffolding, test contamination, and retry budget.