Gradient Clipping Calculator
Prevent exploding gradients · interactive analysis
Recommended thresholds:
RNN 1.0
Transformer 1.0
CNN 5.0
Original global norm—
Clipped global norm—
Scaling factor—
Methodclip_by_norm
When to use each method
- Clip by value – bound each element individually. Use when you want to avoid extreme single values; may distort direction.
- Clip by norm – scale whole gradient if L2 norm > threshold. Preserves direction. Default for RNN, Transformer.
- Global norm – treat all parameters as one vector. Use for multi‑layer / shared parameters.
RNN: 1.0 | Transformer: 1.0 | CNN: 5.0
PyTorch / TF snippets
# PyTorch (clip by norm)
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
# TensorFlow (clip by global norm) tf.clip_by_global_norm(gradients, clip_norm=1.0)
# TensorFlow (clip by global norm) tf.clip_by_global_norm(gradients, clip_norm=1.0)
Snippet updates with selected method & threshold.