MFU calculator
Enter your training throughput in tokens per second and this page computes model FLOPs utilization (MFU), the ratio of achieved to theoretical peak compute during LLM training, using the 6N rule from the PaLM paper, with an optional attention term that grows with sequence length. With the PaLM paper inputs, the calculator gives 45.7% without attention and 46.2% with attention for PaLM 540B, as the paper prints. For Megatron-Turing NLG 530B it gives 29.8% and 30.3%. The paper prints 29.7% and 30.2%. The exact values from the printed inputs are 29.77% and 30.27%. All arithmetic runs in your browser.
Updated 29 October 2026
By Michael Lip, who builds open-source ML tools and developer utilities at zovo.one. The PaLM paper defines model FLOPs utilization as achieved throughput divided by the theoretical peak R = P / (6N + 12LHQT), where P is the total theoretical peak matmul throughput of all chips, N the parameter count, and the 12LHQT term accounts for the dense self-attention FLOPs. Enter your observed tokens per second, model shape and hardware below and the page computes both the plain 6N MFU and the attention-corrected MFU, plus the theoretical peak throughput, the throughput for a target MFU and the days to train D tokens. The page does not send your inputs to a server.
Compute MFU from training throughput
Published MFU rows and the values from their inputs
The throughput, chip count, peak TFLOP/s and paper MFU of both rows come from the PaLM paper (arXiv 2204.02311, Section 4.1, Table 3 and Appendix B). The PaLM 540B shape comes from PaLM Table 1 and Section 5. The MT-NLG shape comes from Smith et al. (arXiv 2201.11990, Section 3.2). MT-NLG 530B ran on 2240 A100 GPUs at 312 peak matmul TFLOP/s and reached 65.43K tokens/s. PaLM 540B ran on 6144 TPU v4 chips at 275 TFLOP/s and reached 238.3K tokens/s. The calculator gives 45.7% and 46.2% for PaLM 540B. For MT-NLG it gives 29.8% and 30.3%, because the exact values are 29.77% and 30.27%. The paper prints 29.7% and 30.2%. These printed values are the exact values rounded down to one decimal. The paper does not state its rounding rule.
| Model | Params N | Layers L | Heads H | Head dim Q | Seq T | Throughput | Chips | Peak TFLOP/s | Paper MFU (no attn / attn) | Exact value from these inputs (no attn / attn) |
|---|---|---|---|---|---|---|---|---|---|---|
| Megatron-Turing NLG 530B | 530B | 105 | 128 | 160 | 2,048 | 65,430 tok/s | 2,240 A100 | 312 | 29.7% / 30.2% | 29.77% / 30.27% |
| PaLM 540B | 540B | 118 | 48 | 256 | 2,048 | 238,300 tok/s | 6,144 TPU v4 | 275 | 45.7% / 46.2% | 45.70% / 46.20% |
MFU values in the PaLM paper
PaLM Table 3 prints these MFU values. GPT-3 175B on V100 GPUs has 21.3%. Gopher 280B on 4096 TPU v3 chips has 32.5%. Megatron-Turing NLG 530B on 2240 A100 GPUs has 30.2%. PaLM 540B on 6144 TPU v4 chips has 46.2%. The GPT-3 and Gopher values are quotes, because the paper does not give all of their inputs. The paper says that PaLM gets high accelerator utilization because of its parallelism strategy, XLA TPU compiler optimizations and the use of "parallel layers". The hardware FLOPs utilization of PaLM, which includes rematerialization FLOPs, is 57.8%. Use the dense peak FLOP/s for the data type that you train in. The NVIDIA A100 datasheet gives 312 TFLOPS BFLOAT16 and 624 TFLOPS with sparsity. If you use the sparsity value, the calculated MFU is half of the correct value.
Formula notes
Per the PaLM paper: an N-parameter decoder-only model requires 6N matmul FLOPs per token seen, 2N for the forward pass and 4N for the backward, because each forward matmul gets two backward matmuls (following Kaplan et al., 2020). Dense self-attention adds 6LH(2QT) FLOPs per token, where L is layers, H heads, Q head dimension and T sequence length; equivalently 12LHQT. MFU is achieved tokens/second divided by the theoretical peak R = P / (6N + 12LHQT), where P is total peak matmul FLOPs/s across the whole fleet. The attention term matters more for small models and for long sequences; at PaLM 540B scale it moved the answer only from 45.7% to 46.2%. The time to train D tokens is D × (6N + 12LHQT) / (P × MFU). This equals D divided by the observed tokens per second. It counts model FLOPs only. PaLM Appendix B says that its compute figures do not account for development, evictions or other inefficiency. The A100 and TPU v4 peaks are the values that the PaLM paper used. The NVIDIA H100 and H200 pages give 1,979 TFLOPS BFLOAT16 with sparsity. The AMD MI300X page gives 989.4 TFLOPS BF16 dense for the H100 SXM (footnote MI300-18). The calculator uses 989.4 for the H100 and half of 1,979, which is 989.5, for the H200. NVIDIA gives the same 1,979 value for the H100 SXM and the H200 SXM. The AMD MI300X page gives 1,978.9 for the H100 SXM. Thus 989.5 is a rounded value, and the 0.1 difference from 989.4 comes from rounding. The AMD MI300X page gives 1,307.4 TFLOPS BF16 dense and 2,614.9 with sparsity. Check the datasheet for the data type that you train in.
Where these numbers come from
All sources were fetched on 2026-10-29. The MFU formula, the 6N rule, the attention term and both reference rows come from Chowdhery et al., "PaLM: Scaling Language Modeling with Pathways", arXiv 2204.02311 version 5, Section 4.1, Table 3 and Appendix B. PaLM 540B has 118 layers, 48 heads and a head size of 256 (Table 1), and the sequence length is 2048 (Section 5). The MT-NLG shape (105 layers, hidden size 20480, 128 heads, sequence length 2048) and the iteration time of 60.1 seconds at batch size 1920 on 280 DGX A100 servers come from Smith et al., arXiv 2201.11990, Sections 2.4 and 3.2. The head dimension is 20480 / 128 = 160. The 6N rule follows Kaplan et al., arXiv 2001.08361. The peak values per chip come from Google Cloud TPU v4 (275 teraflops bf16), the NVIDIA A100 datasheet (312 TFLOPS BFLOAT16, 624 with sparsity), the NVIDIA H100 page, the NVIDIA H200 page and the AMD MI300X page. The results are calculations from your inputs. They are not benchmarks. The arithmetic runs in this browser tab.