When training one model at a time, over a long period of time, we can manually decrease $\alpha$ whenever the learning rate slows down. This requires manually tuning the value of $\alpha$ by periodically checking it over the entire learning period and can be strenuous. So, this method works with training only a small number of models.

University of Michigan - Ann Arbor

Learning rate decay is the gradual reduction of the learning rate as a function of time to speed up the learning algorithm. Decaying the learning rate as the gradient descent approaches completion reduces noise and facilitates a tighter convergence to a target.

Learning Rate Decay

https://www.coursera.org/learn/deep-neural-network?specialization=deep-learning

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

First, we take a constant learning rate represented by the blue line. We see that, as we iterate, the steps are large and noisy and do not converge on a minimum. Instead, it wanders around the minimum.

Next, we take a decaying learning rate represented by the green line. At the start, the learning rate takes large steps with each iteration. But the learning rate is reduced or decayed as it approaches the minimum. This slower learning rate takes smaller tighter steps around the minimum and is closer to convergence.

This method allows us to have relatively fast learning during the initial phases with large steps, but also converge to a minimum during the final phases with slower learning rates and smaller steps.

Example Using Mini-Batch Gradient Descent (Learning Rate Decay)

$$\alpha = \frac{1}{1 + decay\_rate * epoch\_num}\alpha_0$$
, where $\alpha$ is the learning rate in the current epoch, $\alpha_0$ is the initial learning rate, $epoch\_num$ is the current epoch and $decay\_rate$ is the decay rate selected. The decay rate is a tunable hyperparameter.

Initializing $decay\_rate = 1$ and $\alpha_0 = 0.2$, we graph an example with $epoch\_num$ on x-axis and $\alpha$ on y-axis. In the graph we observe the decay of learning rate.

Common Learning Rate Decay Implementation

Exponential Decay: $\alpha =x^{epoch\_num}\cdot \alpha _0$, where $x < 1$

Other formulas: 
$\alpha =\frac{k}{\sqrt{epoch\_num}}\cdot \alpha _0$, where $k$ is a constant hyperparameter
$\alpha =\frac{k}{\sqrt{t}}\cdot \alpha _0$, where $t$ is a mini-batch number

Other Learning Rate Decay Implementations

Manual Implementation Learning Rate Decay

Learning rate is a positive scalar determining the size of the step in method of steepest descent. 

Learn Before

Related