Learning Rate Finder

How this page computes its curve

Read this first, because it changes how you should use the numbers above. The curve on this page is generated, not measured. It comes from a closed-form function with a fixed base loss per loss type, plus a small random noise term. It never sees your model, your data, or your optimizer.

So treat the four readouts as a worked illustration of what a range test looks like and how to read one. They are not a result for your network. To get a number that applies to your training run, copy the snippet below and run the real sweep. The shape you see here is what you should expect to see there.

import torch

# Real LR range test. Swap in your own model, optimizer and loader.
lo, hi, steps = 1e-7, 1.0, 200
mult = (hi / lo) ** (1 / steps)
lr = lo
opt = torch.optim.SGD(model.parameters(), lr=lr, momentum=0.9)
lrs, losses, ema, beta = [], [], 0.0, 0.9

for i, (x, y) in enumerate(loader):
    if i >= steps:
        break
    for g in opt.param_groups:
        g['lr'] = lr
    opt.zero_grad()
    loss = criterion(model(x), y)
    loss.backward()
    opt.step()
    ema = beta * ema + (1 - beta) * loss.item()
    losses.append(ema / (1 - beta ** (i + 1)))   # bias-corrected
    lrs.append(lr)
    lr *= mult

# Take the rate where the smoothed curve falls fastest,
# not where it bottoms out. The minimum is usually already unstable.

What a learning rate range test does

Leslie Smith introduced the test in Cyclical Learning Rates for Training Neural Networks (2015). You run one short training pass, raising the learning rate from very small to very large, and record the loss at every step. One run, one curve, no grid search.

That curve has three parts. While the rate is too small the loss barely moves, because the updates are tiny and progress is impractically slow. Then the loss drops steeply. That middle band is large enough to learn and small enough to stay stable, and it's the part you care about.

Past it, the loss climbs or swings wildly. Updates now overshoot the minima and the run diverges. Pick your rate from the steepest part of the descent, a little below the point of fastest drop, so you keep the speed and leave yourself a margin.

Why the learning rate matters

It sets the step size of every gradient update. Too small and training crawls or settles somewhere poor. Too large and it diverges. Width, depth and regularisation change what a model can represent, but the learning rate decides whether training works at all.

The right value moves with the loss surface, the batch size, the architecture, the optimizer, and how far into training you are. What suits a ResNet-50 on ImageNet won't suit a Transformer on text. That's the argument for measuring it rather than reaching for a default, since the test probes your own model on your own data.

Batch size is the coupling people miss. Goyal et al. showed in Accurate, Large Minibatch SGD that multiplying the batch by k and the learning rate by that same k holds ImageNet accuracy out to very large batches, given a short warmup.

Scaling by the square root of k is the more cautious habit. Either way, a rate you measured at one batch size doesn't carry over unchanged to another.

Smoothing a noisy loss curve

Raw training loss jumps around because every mini-batch holds different samples. That noise hides the trend you're trying to read. An exponential moving average fixes it by weighting recent steps more heavily than old ones.

Beta sets the strength. At 0.0 you see the raw loss, and at 0.99 the curve is heavily damped. Somewhere between 0.8 and 0.95 usually reads well, though that band is a working habit rather than a published result.

Too little smoothing leaves the curve unreadable. Too much introduces lag, which drags the apparent best point toward rates that are actually too high. Dividing by 1 minus beta to the power t removes the bias the average carries over its first few steps.

One-cycle against warm restarts

Once you have a rate, you need a schedule. Two of them hold up well.

One-cycle comes from Smith's A Disciplined Approach to Neural Network Hyper-Parameters (2018). The rate climbs to a maximum, then anneals back toward zero over the rest of the run. Smith puts the lower bound a factor of 10 or 20 below that maximum when only one cycle is used. Momentum runs the opposite way across the same cycle, roughly 0.95 down to 0.85 and back up.

Paired with a large maximum rate this gives what Smith named super-convergence, training networks an order of magnitude faster than standard schedules.

Warm restarts come from SGDR (Loshchilov and Hutter, 2016). Cosine annealing walks the rate down to near zero, then snaps it back up and repeats, which gives the optimizer a chance to leave a basin it has settled into. Each period either stays fixed or grows by a factor after every cycle.

The paper reports comparable results in 2 to 4 times fewer epochs than the schedule it replaced. Checkpoint at the end of each cycle and you get snapshot ensembling nearly for free.

Training one model, start with one-cycle. Building an ensemble, warm restarts earn their keep.

Other schedules worth knowing

Step decay cuts the rate by a fixed factor at set epochs. PyTorch's own ImageNet example still ships StepLR(optimizer, step_size=30, gamma=0.1), a tenfold cut at epochs 30, 60 and 90. It's old, and it still competes.

Exponential decay multiplies by a constant below 1 every step or epoch. Polynomial decay follows a curve you shape with a power parameter. ReduceLROnPlateau watches a validation metric and only cuts when progress stalls, so it reacts to the run rather than to a calendar.

Adam, AdamW and LAMB adapt a rate per parameter, which softens how much the global value matters. It doesn't remove it. The paper to read is Loshchilov and Hutter's Decoupled Weight Decay Regularization (ICLR 2019). Separating weight decay from the gradient update improved how well Adam generalised, enough to compete with SGD plus momentum, which it had usually lost to.

Running the test on your own model

• Run the range test on your full dataset (or a representative subset) with the exact batch size and optimizer you plan to use for training. These factors affect the optimal LR.
• Use 100-300 steps covering 5-6 orders of magnitude. Starting at 1e-7 and ending at 1-10 usually covers the useful range for most architectures and optimizers.
• Apply EMA smoothing with beta=0.9 to see the trend clearly. Adjust if the curve is still too noisy or overly smoothed.
• The optimal LR is at the steepest descent, not at the minimum loss. The minimum is often already in the instability zone.
• For transfer learning, the optimal LR for fine-tuning is typically 10-100x smaller than for training from scratch, because the pretrained weights are already near a good minimum.
• Repeat the test 2-3 times with different random seeds to verify the optimal range is consistent. If results vary widely, use the more conservative (lower) estimate.
• When using learning rate warmup (recommended for Transformers and large batch training), the LR range test gives you the peak learning rate. The warmup period typically covers 5-10% of total training steps.

Related tools

About the author

Michael Lip builds open-source ML tools and developer utilities at zovo.one. ml0x is part of the Zovo Tools network, a collection of free, privacy-first tools for developers and data scientists. No tracking, no accounts required, no data leaves your browser.