Plug in your architecture and get parameter count, training/inference FLOPs, KV-cache size, and a full VRAM breakdown for training — instantly, in the browser.
This calculator turns the standard transformer scaling formulas into a live estimator. Set the number of layers, model width, attention heads, sequence length, batch size and precision, then read off how big the model is, how much floating-point work a forward/backward pass costs, and whether it will fit in your GPU. Everything runs client-side — no data leaves your machine.
Params use 12·L·d_model² for blocks plus token & positional embeddings. FLOPs follow the standard ~6N training / ~2N inference rule per token, with exact attention and FFN terms added.
Per training step over batch×seq tokens. A100 (312 TFLOP/s bf16) reference time shown below.
A decoder-only transformer is dominated by its repeated blocks. Each block holds four attention projection matrices (query, key, value, output) of size d_model² and a two-layer feed-forward network of size 2·ffn·d_model². With the usual ffn = 4, one block is 12·d_model² weights, so all blocks total N_blocks = 12·L·d_model². Embeddings add (vocab + seq)·d_model for the token and positional tables, which matter a lot for small models and almost nothing for large ones.
For compute we use the well-known accounting from the Kaplan and Chinchilla scaling papers: a forward pass costs about 2 FLOPs per parameter per token (one multiply, one add), and the backward pass roughly doubles that, giving the famous 6·N training rule. Attention over a length-S sequence adds a term that grows as S², which is why long-context training gets expensive fast even when the parameter count is fixed. This tool separates that quadratic attention cost from the linear FFN/projection cost so you can see when context length — not model size — becomes your bottleneck.
Memory has four training buckets: the weights themselves, an equal-sized gradient tensor, the optimizer states (Adam keeps two extra copies at master precision), and activations saved for the backward pass. Activation memory scales with batch · seq · d_model · L plus the batch · heads · seq² attention-score buffer, which is the term that explodes with long sequences and is exactly what FlashAttention and activation-checkpointing target. For inference, the relevant number is instead the KV-cache: 2·L·seq·d_model bytes per sequence in the batch, holding the keys and values that autoregressive decoding reuses. Slide the inputs above and every figure recomputes live.