Pick a 2D test loss, set a starting point, and run each optimizer's exact update rule side by side. The trajectory plot shows the path each takes across the contours; the loss curve shows how fast each one converges. Every buffer — momentum velocity, Adam's bias-corrected moments, RMSprop's running variance — is computed live so you can watch learning rate, β1, β2 and ε reshape the descent.
Start: click the contour plot to set init point.
| Optimizer | final loss | x | y | ↓ from start | diverged? |
|---|
Optimizers differ only in how they turn a gradient into a step. Plain SGD steps straight down the gradient, so on the curved Rosenbrock valley it zig-zags across the walls and creeps along the floor — the classic ill-conditioning problem. Momentum accumulates a velocity vector v: past gradients keep pushing it forward, damping the oscillation and accelerating along the valley floor, at the cost of overshooting near the minimum when β1 is high.
RMSprop attacks the same problem differently. It keeps a running average s of squared gradients and divides each step by √s + ε, so a coordinate with historically large gradients gets a smaller effective learning rate. This per-parameter scaling is why RMSprop tolerates a single global learning rate across dimensions of wildly different curvature. Adam combines both ideas: a momentum-like first moment m and an RMSprop-like second moment v. Because both start at zero they are biased toward zero early on, so Adam applies the bias correction m̂ = m/(1−β1t) and v̂ = v/(1−β2t) — the term that makes Adam take confident steps from iteration one instead of stalling.
Try this: on Rosenbrock, set learning rate near 0.01. SGD barely moves, Momentum overshoots and loops, while Adam and RMSprop track the valley smoothly. Now push ε up toward 1e-2 and watch Adam degrade toward SGD-like behavior, because the denominator stops adapting. Drop β2 to 0.9 and the second moment forgets faster, making the adaptive step noisier. The gradients here are computed analytically, and the same buffers PyTorch and TensorFlow maintain internally are exposed in the table, so the paths you see match what a real training loop would trace on these surfaces.