Tokens per second calculator

The calculator

Ceiling:

tokens per second. Upper limit, not a measured speed.

Bytes read per token:

.

The formula and where it comes from

At batch 1 the decoder reads nearly every weight once per token, plus the KV cache of the current context. The exception is the input embedding table, because decode reads only one row of it per token. Qwen2.5-7B-Instruct sets "tie_word_embeddings": false and "vocab_size": 152064 in its config.json, so its input embedding table holds 152064 x 3584 = 544997376 of its 7615616512 parameters, about 7.2 percent. This page counts the whole GGUF file, so for a model with untied embeddings the printed ceiling is slightly below the true bandwidth limit. So the ceiling is

ceiling tokens/s = memory bandwidth in bytes per second / (weight bytes + KV bytes per token x context tokens).

GPU memory bandwidth figures come from NVIDIA. The H100 product page lists GPU Memory Bandwidth 3.35TB/s for H100 SXM and 3.9TB/s for H100 NVL. The A100 product page lists 1,935 GB/s for A100 80GB PCIe and 2,039 GB/s for A100 80GB SXM. The 1,555 GB/s figure for A100 40GB PCIe and A100 40GB SXM is from the NVIDIA A100 datasheet (PDF).

The source for memory-bound decode at small batch is Pope et al., Efficiently Scaling Transformer Inference, which states that "At small batch sizes and sequence lengths, the time to load weights dominates" (full text on ar5iv). The KV bytes per token come from each model's own config.json. For Qwen2.5-7B-Instruct, config.json sets "num_hidden_layers": 28 and "num_key_value_heads": 4, and "hidden_size": 3584 divided by "num_attention_heads": 28 gives a head dim of 128. With 2 bytes at fp16 for both K and V that is 2 x 28 x 4 x 128 x 2 = 57344 bytes per token (config.json). Weight bytes are the summed part sizes of Qwen's published GGUF files on the Hugging Face tree API. Parameter counts come from the safetensors.total field on each model's model-info API record, which reports 7615616512 for the 7B.

A worked example

Qwen2.5-7B-Instruct, q4_k_m, H100 SXM, context 0. The q4_k_m GGUF is split into two parts of 3993201344 and 689872288 bytes, 4683073632 bytes total (GGUF tree). Divide the H100 SXM bandwidth of 3350 GB/s, that is 3,350,000,000,000 bytes per second, by those weight bytes and the ceiling is about 715.3 tokens per second at an empty context. With the 8098525408 byte q8_0 file the same GPU tops out at about 413.7 tokens per second, and the 15237853536 byte fp16 file gives about 219.8. All three ceilings come from measured file sizes rather than a nominal bytes-per-parameter guess.

How the ceiling drops as context grows

Every token in the context adds KV bytes to the per-token read. For the 7B that is 57344 bytes per token at fp16. The table below is written into the page itself, so you can read it without running anything. Each cell is bandwidth divided by weight bytes plus KV bytes at that context length, computed once from the same fields the calculator uses and written into the page as static numbers.

Each Qwen2.5 config.json sets "max_position_embeddings": 32768. The Qwen2.5-7B-Instruct model card says that the 7B reaches 131,072 tokens only when YaRN rope scaling is added to config.json, and the 0.5B, 1.5B and 3B model cards give a full context of 32,768 tokens. The 65536 and 131072 columns assume YaRN.

A 131072 token context adds the same 7516192768 bytes of KV cache to each token read in every row. The fp16 row starts lowest because its file is about 15.24 GB, so the KV term is a smaller share of its read and its ceiling falls by about a third (219.8 to 147.2). The q4_k_m ceiling falls by about 62 percent (715.3 to 274.6).

What the ceiling does not include

Kernel efficiency, batching, prefill, sampling, and CPU overhead all sit below this roofline. A real single-stream decode number will be lower than what the calculator prints, and a batched one can be higher per GPU while staying lower per stream. Treat the output as a planning bound and check it against your own measurement. Pope et al. (arXiv 2211.05102) frame the small-batch regime as bandwidth bound for the same reason this page does.

How we know the numbers are right

Methodology. Every GGUF weight byte total is the sum of every split part of the published file, read from the Hugging Face tree API. Every parameter count is the safetensors.total field from the model-info API. Every KV bytes-per-token figure is computed from that model's config.json with the formula above. Nothing here is estimated from a nominal bytes-per-parameter constant when a measured file size exists, and the browser recomputes every result from the inputs you enter. No data leaves your machine.

A note on the architecture behind the KV term. Qwen2.5-7B-Instruct uses grouped query attention, 28 attention heads sharing 4 key value heads per the config's "num_key_value_heads": 4 (config.json). A calculator that used num_attention_heads instead would over-report the KV cache by a factor of 7 (28 / 4). The figures here use num_key_value_heads, as the cache does.

Other tools that do different work

MFU calculator takes a measured throughput and converts it to training compute utilization. LLM memory calculator sizes total memory for training or inference. Quantization calculator compares GGUF file sizes across levels.