Learn Before
Formula

Common Learning Rate Decay Formula

α=11+decay_rateepoch_numα0\alpha = \frac{1}{1 + decay\_rate \cdot epoch\_num} \alpha_0, where α\alpha is the learning rate in the current epoch, α0\alpha_0 is the initial learning rate, epoch_numepoch\_num is the current epoch, and decay_ratedecay\_rate is the selected decay rate. The decay rate is a tunable hyperparameter. Initializing decay_rate=1decay\_rate = 1 and α0=0.2\alpha_0 = 0.2, we can graph an example with epoch_numepoch\_num on the x-axis and α\alpha on the y-axis to observe the decay of the learning rate.

Image 0

0

2

Updated 2026-06-16

Tags

Data Science