Speculative decoding speedup calculator

By Michael Lip, who builds open-source ML tools and developer utilities at zovo.one. Enter an acceptance rate α, a draft length γ, a cost coefficient c and an operations ratio ĉ. The page applies the formulas of Leviathan et al. (2023) to give the expected number of tokens generated per run of the algorithm, the expected walltime improvement factor over standard decoding, the expected factor of increase in total arithmetic operations, and the draft length from 1 to 20 that gives the largest improvement. All arithmetic runs in this browser tab.

Compute a plan

Inputs Acceptance rate α, between 0 and 1 (for example 0.8). The chance that the target model accepts one draft token. Draft length γ, a whole number of tokens per step, 1 to 1000 (for example 5). Relative cost of one draft token c, 0 or more (for example 0.05). Time of one draft forward pass divided by the time of one target forward pass. Operations ratio ĉ, 0 or more (for example 0.05). Arithmetic operations per token of M_q divided by those of M_p (Definition 3.10). Used only for the operations result.

Formulas

The page uses the notation of Leviathan et al. (2023): M_p is the target model, M_q is the draft model, γ is the number of tokens the draft model writes per step, α is the expected acceptance rate of a single token, and c is the relative cost of one draft forward pass. Chen et al. (2023) use different letters for the same quantities, K for the draft length and p for the draft model, and this page states that mapping once here.

expected tokens per step = E(tokens) = (1 − α^γ+1) / (1 − α) (Eq. 1 of arXiv 2211.17192)

expected speedup = (1 − α^γ+1) / ((1 − α) × (γc + 1)) (Theorem 3.8)

best γ for a given α and c = the γ in 1 to 20 with the largest Theorem 3.8 value, compared unrounded

Corollary 3.9: if α > c, a γ with an improvement exists, and the improvement factor is at least (1 + α) / (1 + c) (Theorem 3.8 at γ = 1)

expected factor of increase in total arithmetic operations = (1 − α)(γĉ + γ + 1) / (1 − α^γ+1) (Theorem 3.11, ĉ from Definition 3.10)

At α = 1 the page uses the limits γ + 1 tokens per step, speed (γ + 1) / (γc + 1) and operations (γĉ + γ + 1) / (γ + 1), because Eq. 1 and Theorem 3.11 divide by zero there. With c = 0 the Theorem 3.8 speed rises toward the bound 1 / (1 − α) as γ grows, so the best-γ search ends at the search cap of 20 and the page says that the cap set the answer. When α is less than or equal to c, no γ gives a speedup above 1 and the page says so instead of showing a best γ as a gain.

Table 1 of the paper

Table 1 of Leviathan et al. (2023) gives the total number of arithmetic operations and the inference speed against the baseline, for six pairs of α and γ, with c = ĉ = 0. Table 1 is a calculation from the formulas, not a measurement. The columns are the paper's operations value (Theorem 3.11), the paper's speed value (Theorem 3.8) and this page's value of the speed formula. All six rows agree to two decimals.

α	γ	Operations (paper)	Speed (paper)	Speed (this page)
0.6	2	1.53	1.96	1.96
0.7	3	1.58	2.53	2.53
0.8	2	1.23	2.44	2.44
0.8	5	1.63	3.69	3.69
0.9	2	1.11	2.71	2.71
0.9	10	1.60	6.86	6.86

Table 1 is not a measurement. Table 4 of the same paper compares Theorem 3.8 with measured T5-XXL walltimes: the measured value is higher in 8 of 12 rows, equal in 1 row and lower in 3 rows. The paper gives two reasons for the differences: optimization differences between its implementation and the baseline, and the i.i.d. assumption for the acceptance of each token, which is only an approximation. For English to German translation with T5-XXL as M_p and a bigram model as M_q, the paper gives α ≈ 0.2 and, with c = 0, an inference speed improvement factor of 1.25X with γ = 3 (Section 3.6). In the paper's experiments, where M_q was typically a couple of orders of magnitude smaller than M_p, c was always less than 0.05. Table 4 has c = 0.11 for T5-LARGE, so c is not always that small.

Worked examples

Reproducing Table 1. With α = 0.8 and γ = 5 and c = 0, Eq. 1 gives (1 − 0.8⁶) / (1 − 0.8) = 3.68928 tokens per step, and Theorem 3.8 gives the same 3.69 speedup shown in the table, since the draft is free.

Bigram draft. With α = 0.2, γ = 3 and c = 0, Eq. 1 and Theorem 3.8 both give 1.248, which rounds to the 1.25X value in Section 3.6 of the paper for English to German translation.

Best γ with a paid draft. With α = 0.8 and c = 0.05, the best γ from 1 to 20 is 8, with speed 3.092080. With α = 0.6 and c = 0.1, the best γ is 3, with speed 1.673846.

Corollary 3.9 check. With α = 0.8, c = 0.05 and γ = 1, Theorem 3.8 gives (1 + α) / (1 + c) = 1.714286, exactly the corollary's bound for a one-token draft.

Free draft cap. With α = 0.9 and c = 0, the search hits the γ = 20 cap with speed 8.905810 against the bound 1 / (1 − α) = 10. The page reports that the cap set the answer.

Near tie. With α = 0.7 and c = 0.05, γ = 6 gives 2.3529377 and γ = 5 gives 2.3529360. The page compares unrounded values, picks γ = 6, and notes that the neighbour is within 0.0001. Rounded to two decimals both show 2.35.

No gain. With α = 0.5 and c = 0.6, no γ beats plain target decoding, because the draft costs more per token than the acceptance rate justifies. The page states this instead of a speedup.

Measured values from Chen et al. (2023), arXiv 2302.01318v1. Setup: Chinchilla 70B as the target model, a 4B draft model, K = 4 (K is Leviathan γ), batch size 1, 16 TPU v4. The abstract reports a 2 to 2.5× decoding speedup in a distributed setup. Table 1 gives 1.92× on XSum with nucleus sampling, 2.01× on XSum with greedy sampling and 2.46× on HumanEval with nucleus sampling, against a baseline of 14.1 ms per token. The draft model samples at 1.8 ms per token. These are measured values. Chen et al. do not report α in Table 1, so this page does not use them to test the formulas, and it does not compute speedups from the millisecond values.

Limits

The results are expected values from the formulas. They are not measured speeds. Theorem 3.8 assumes that the acceptance of each draft token is i.i.d. with one constant rate α, that the hardware can run γ + 1 evaluations of M_p in parallel with no increase in walltime, and that the generations are long enough (the paper notes that the improvement is capped by the number of generated tokens). The page does not model batch size, memory bandwidth, cost or prices, and it does not predict the speedup of a named model, API or GPU. The best-γ search covers γ = 1 to 20.

Sources

All sources were fetched on 2026-10-19. Leviathan, Yaniv et al. (2023), "Fast Inference from Transformers via Speculative Decoding", arXiv 2211.17192 version 2: Section 3.1 for α = E(β), Definition 3.7 for c, Eq. 1 for expected tokens, Theorem 3.8 for the expected speedup, Corollary 3.9 for the one-token bound, Theorem 3.11 for expected operations, Definition 3.10 for ĉ, Section 3.5 for the choice of γ, Section 3.6 for the c = 0 bound and the bigram example, Table 1 for the operations and speed values with c = ĉ = 0, Table 4 and Appendix A.3 for the comparison with measured walltimes, and the text after Definition 3.7 for the cost caveat. Chen, Charlie et al. (2023), "Accelerating Large Language Model Decoding with Speculative Sampling", arXiv 2302.01318 version 1: abstract for the 2 to 2.5× range, Table 1 for the measured speedups at batch size 1 with K = 4, and the Results section for the 4B draft model, 16 TPU v4 and the 1.8 ms and 14.1 ms per token values. Links: arXiv 2211.17192, arXiv 2302.01318.