When training a single model over an extended period, one can manually decrease the learning rate ($$\alpha$$) whenever the learning process slows down. This approach requires periodically checking the model's progress to manually tune $$\alpha$$, which can be strenuous and is typically practical only when training a small number of models.

University of Michigan - Ann Arbor

Google

Learning rate decay is the gradual reduction of the learning rate as a function of time to speed up the learning algorithm. Decaying the learning rate as the gradient descent approaches completion reduces noise and facilitates a tighter convergence to a target.

Learning Rate Decay

https://www.coursera.org/learn/deep-neural-network?specialization=deep-learning

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

First, we take a constant learning rate represented by the blue line. We see that, as we iterate, the steps are large and noisy and do not converge on a minimum. Instead, it wanders around the minimum.

Next, we take a decaying learning rate represented by the green line. At the start, the learning rate takes large steps with each iteration. But the learning rate is reduced or decayed as it approaches the minimum. This slower learning rate takes smaller tighter steps around the minimum and is closer to convergence.

This method allows us to have relatively fast learning during the initial phases with large steps, but also converge to a minimum during the final phases with slower learning rates and smaller steps.

Example Using Mini-Batch Gradient Descent (Learning Rate Decay)

$$\alpha = \frac{1}{1 + decay\_rate * epoch\_num}\alpha_0$$
, where $\alpha$ is the learning rate in the current epoch, $\alpha_0$ is the initial learning rate, $epoch\_num$ is the current epoch and $decay\_rate$ is the decay rate selected. The decay rate is a tunable hyperparameter.

Initializing $decay\_rate = 1$ and $\alpha_0 = 0.2$, we graph an example with $epoch\_num$ on x-axis and $\alpha$ on y-axis. In the graph we observe the decay of learning rate.

Common Learning Rate Decay Implementation

Exponential Decay: $\alpha =x^{epoch\_num}\cdot \alpha _0$, where $x < 1$

Other formulas: 
$\alpha =\frac{k}{\sqrt{epoch\_num}}\cdot \alpha _0$, where $k$ is a constant hyperparameter
$\alpha =\frac{k}{\sqrt{t}}\cdot \alpha _0$, where $t$ is a mini-batch number

Learn Before

Related