KV cache calculator

Size the cache for your model

All computation runs client side. No data leaves your browser.

The formula, and where it comes from

kv_bytes = 2 (one K tensor, one V tensor)
         × bytes_per_element
         × num_hidden_layers
         × num_key_value_heads
         × head_dim
         × context_length
         × batch_size

Two terms in that product are the ones people get wrong. The head count is num_key_value_heads, because grouped-query attention uses an intermediate number of key-value heads, more than one and fewer than the number of query heads (Ainslie et al., GQA, arXiv 2305.13245). And the reason the cache matters at all is bandwidth, not capacity. Multi-query attention was introduced to cut the size of the keys and values tensors loaded during incremental decoding (Shazeer, Fast Transformer Decoding, arXiv 1911.02150), which is what makes cache size the bottleneck this tool estimates.

The other term is head_dim. Read it from the file when the file has it. Deriving it from hidden_size divided by num_attention_heads is a habit that breaks on real configs, and two of the models below are the proof.

Estimates use standard scaling formulas. Real workloads vary with fused kernels, sharding, and precision.

Every model we fetched, in one table

Bytes per token, KiB per token, and the cache at a 128k context, computed at bf16 from each model's published config.json. The last two columns are the part you will not find on the pages currently ranking for this term. Query heads per KV head is the grouped-query ratio. Naive over-report is how much larger a calculator's answer gets when it multiplies by num_attention_heads instead.

^† Phi-3-mini-4k-instruct and OLMo-2-1124-7B each declare max_position_embeddings of 4,096, and SmolLM2-1.7B declares 8,192. For those three the 128k column is an extrapolation past the model's own published position-embedding limit, not a supported configuration. The row is there so the shape of the cache is comparable, and it is labelled so you do not mistake it for a spec.

Three rows in that table have a query-heads-per-KV-head ratio of one. Phi-3-mini-4k-instruct, OLMo-2-1124-7B and SmolLM2-1.7B each set num_key_value_heads equal to num_attention_heads, which is full multi-head attention with no grouped-query sharing. That is why their per-token cache is large relative to their parameter count, and it is also why their over-report column reads zero. Nothing to over-report when there is no sharing to miss.

Two configs where the published head_dim disagrees with the derivation

Across the twenty-one model configs fetched for this page, two publish a head_dim that does not equal hidden_size divided by num_attention_heads. Both break a calculator that derives instead of reads.

Qwen3-32B publishes head_dim of 128 while its hidden_size of 5120 divided by its num_attention_heads of 64 gives 80. Deriving there understates the cache by 37.5 percent. Mistral NeMo Base 2407 runs the other way. It publishes head_dim of 128 while its hidden_size of 5120 divided by its num_attention_heads of 32 gives 160, so deriving overstates by 25 percent. The explicit field is authoritative in both cases. Llama 3.2 1B, Qwen3-8B and Qwen3-14B also publish an explicit head_dim, so read it there too.

Architectures this formula refuses

Refusing beats approximating. A calculator that quietly substitutes a plausible value is worse than one that stops, because you cannot see the error in a number that looks reasonable.

Falcon-7B sets "multi_query": true in its config and carries no num_key_value_heads key at all. Multi-query attention shares a single key-value head across all of the attention heads, so a calculator reading num_key_value_heads must refuse the model rather than silently substituting num_attention_heads. Falcon's config does declare 71 attention heads and hidden_size 4544, which yields a 64-wide head, but the cache holds one shared key-value head, not seventy-one, and the file never states that count.

DeepSeek-V2-Lite declares "kv_lora_rank": 512, the signature of multi-head latent attention, which caches a compressed latent vector instead of per-head keys and values. It also declares qk_rope_head_dim of 64 alongside qk_nope_head_dim of 128, a decoupled RoPE split with no counterpart in the grouped-query formula. DeepSeek's authors report that the latent compression reduces the KV cache by 93.3 percent relative to DeepSeek 67B. Layers times KV heads times head_dim does not describe that cache, so the tool refuses it.

Mixture-of-experts is a different story. Mixtral 8x7B declares eight local experts and two experts per token, and its cache still depends only on layers, KV heads and head_dim. The expert feed-forward layers add no KV-cache bytes per token at all.

Where these numbers come from

Every per-token figure here was fetched, not typed. Each one is read out of that model's own published config.json on the Hugging Face Hub, then recomputed in code as two bytes at fp16, times two for the K and V tensors, times num_key_value_heads, times head_dim, times num_hidden_layers.

Each figure is then cross-checked against ModelScope, an independent host run by a different operator. Qwen2.5-7B is the worked example. ModelScope serves a config.json byte-identical in its architecture fields to the Hugging Face copy, both declaring 28 layers, 28 attention heads and 4 key-value heads, and both declaring hidden_size 3584. A value the two hosts do not agree on is held back and nothing is published for it.

I maintain ml0x.com, and its tool pages compute entirely client side. The confusion matrix calculator and the transformer FLOPs and memory calculator both say so on the page, and I checked that by direct inspection of the live pages on 2026-07-10. This one behaves the same way. Your config never leaves the tab.

This tool is for planning and teaching. Verify against a profiler run before committing a hardware budget.

Questions people ask about KV cache size

Why num_key_value_heads and not num_attention_heads

Grouped-query attention uses an intermediate number of key-value heads, more than one and fewer than the number of query heads. The cache stores one key and one value per KV head per layer, so the KV head count is the correct multiplier. Multiply by the query-head count and your answer is too large by the ratio between them, which is the column this page publishes.

Does mixture-of-experts inflate the cache

No. Mixtral 8x7B declares eight local experts and two experts per token, yet its KV cache depends only on layers, KV heads and head_dim. The expert feed-forward layers add nothing per token.

Why does the tool refuse Falcon

Falcon-7B's config sets multi_query to true and contains no num_key_value_heads key. Multi-query attention shares a single key-value head across all attention heads, so there is nothing in the file for the formula to read, and guessing from num_attention_heads would be a fabricated number.

Why does the tool refuse DeepSeek V2

DeepSeek-V2-Lite declares kv_lora_rank of 512, which means multi-head latent attention. The cache is a compressed latent vector rather than per-head keys and values, so the layers-times-KV-heads-times-head_dim product does not describe it.

Why do three rows show a context longer than the model supports

Phi-3-mini-4k-instruct and OLMo-2-1124-7B each declare max_position_embeddings of 4,096, and SmolLM2-1.7B declares 8,192. Their 128k column is an extrapolation past the published limit, kept for shape comparison and marked with a dagger.

My model is not in the list

Choose the custom option, or paste the raw config.json into the box. The parser reads the same three keys and applies the same refusals.

Limits of this estimate

Weights, activations, fragmentation and paged-allocator overhead are all outside this number. So is any framework that stores the cache in a layout other than dense per-layer K and V tensors. What you get here is the KV cache term alone, at the precision you selected, for the context and batch you entered.

ml0x publishes free machine learning calculators and explainers. Every number on this page is computed in your browser from the inputs you enter. Nothing is sent to a server. Written by Michael Lip, part of the Zovo Tools network.