Learning rate is a positive scalar determining the size of the step in method of steepest descent. 

University of Michigan - Ann Arbor

Learning rate decay is the gradual reduction of the learning rate as a function of time to speed up the learning algorithm. Decaying the learning rate as the gradient descent approaches completion reduces noise and facilitates a tighter convergence to a target.

Learning Rate Decay

The derivative of a function $$f: \mathbb{R} ightarrow \mathbb{R}$$ at a point $$x$$ represents the rate of change in the function with respect to changes in its arguments. Formally, it is defined as the limit of the ratio between a perturbation $$h$$ and the change in the function value as $$h$$ approaches zero: $$f'(x) = \lim_{h ightarrow 0} \frac{f(x+h) - f(x)}{h}$$. The derivative essentially measures how rapidly a function's value would increase or decrease given an infinitesimally small adjustment to its parameter.

Derivative of a Scalar Function

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

First, we take a constant learning rate represented by the blue line. We see that, as we iterate, the steps are large and noisy and do not converge on a minimum. Instead, it wanders around the minimum.

Next, we take a decaying learning rate represented by the green line. At the start, the learning rate takes large steps with each iteration. But the learning rate is reduced or decayed as it approaches the minimum. This slower learning rate takes smaller tighter steps around the minimum and is closer to convergence.

This method allows us to have relatively fast learning during the initial phases with large steps, but also converge to a minimum during the final phases with slower learning rates and smaller steps.

Example Using Mini-Batch Gradient Descent (Learning Rate Decay)

$$\alpha = \frac{1}{1 + decay\_rate * epoch\_num}\alpha_0$$
, where $\alpha$ is the learning rate in the current epoch, $\alpha_0$ is the initial learning rate, $epoch\_num$ is the current epoch and $decay\_rate$ is the decay rate selected. The decay rate is a tunable hyperparameter.

Initializing $decay\_rate = 1$ and $\alpha_0 = 0.2$, we graph an example with $epoch\_num$ on x-axis and $\alpha$ on y-axis. In the graph we observe the decay of learning rate.

Common Learning Rate Decay Implementation

Exponential Decay: $\alpha =x^{epoch\_num}\cdot \alpha _0$, where $x < 1$

Other formulas: 
$\alpha =\frac{k}{\sqrt{epoch\_num}}\cdot \alpha _0$, where $k$ is a constant hyperparameter
$\alpha =\frac{k}{\sqrt{t}}\cdot \alpha _0$, where $t$ is a mini-batch number

Other Learning Rate Decay Implementations

When training one model at a time, over a long period of time, we can manually decrease $\alpha$ whenever the learning rate slows down. This requires manually tuning the value of $\alpha$ by periodically checking it over the entire learning period and can be strenuous. So, this method works with training only a small number of models.

Manual Implementation Learning Rate Decay

Learning Rate

On a straight line, the function's derivative...

Gradient descent is a fundamental optimization algorithm that leverages gradients to minimize a model's loss function. Because the gradient of a function points in the direction of steepest ascent, moving the model's parameters in the opposite direction iteratively lowers the loss. Each step of such gradient-based optimization algorithms requires calculating the exact gradient of the loss with respect to the parameters.

Gradient Descent

https://www.wired.com/2015/04/crash-course-derivatives/

A crash course of derivatives

A derivative of a derivative is known as second derivative. Second derivative is often used to measure curvature.

Second Derivative

Hessian matrix is defined as a matrix containing second derivatives of function having multiple input dimensions. It's defined such that
$H(f)(x)_{i,j} = \frac{\partial^2}{\partial x_i \partial x_j}f(x)$

Hessian Matrix 

Denote the function as $f(x)$, $g$ is the gradient and $H$ is is the Hessian at $x^{(0)}$. We calculate the new point $x = x^{(0)} - \epsilon g$. We can obtian that
$f(x^{(0)} - \epsilon g) \approx f(x^{(0)}) - \epsilon g^Tg + \frac{1}{2} \epsilon^2 g^THg$ 
According to the above equation, the optimal step size when $g^THg$ is positive is
$\epsilon^* = \frac{g^Tg}{g^THg}$

Optimal Step Size according to Taylor Series Approximation

A Lipschitz continuous function is a functionfwhose rateof change is bounded by a Lipschitz constant $L$:
$L = \forall x \forall y. |f(x)-f(y)| \leq L ||x-y||_2$

Lipschitz Continuous

Functions that are composed of other differentiable functions can be differentiated using systematic rules. For differentiable functions $$f(x)$$ and $$g(x)$$ and a constant $$C$$, the primary rules are:
- Constant multiple rule: $$\frac{d}{dx} [C f(x)] = C \frac{d}{dx} f(x)$$
- Sum rule: $$\frac{d}{dx} [f(x) + g(x)] = \frac{d}{dx} f(x) + \frac{d}{dx} g(x)$$
- Product rule: $$\frac{d}{dx} [f(x) g(x)] = f(x) \frac{d}{dx} g(x) + g(x) \frac{d}{dx} f(x)$$
- Quotient rule: $$\frac{d}{dx} \frac{f(x)}{g(x)} = \frac{g(x) \frac{d}{dx} f(x) - f(x) \frac{d}{dx} g(x)}{g^2(x)}$$

Differentiation Rules

Several fundamental functions have well-known derivatives that serve as building blocks for more complex calculus operations. For any constant $$C$$ and any power $$n 
eq 0$$, the standard derivatives are:
- Constant: $$\frac{d}{dx} C = 0$$
- Power: $$\frac{d}{dx} x^n = n x^{n-1}$$
- Exponential: $$\frac{d}{dx} e^x = e^x$$
- Natural Logarithm: $$\frac{d}{dx} \ln x = x^{-1}$$

Derivatives of Common Functions

For functions of a single variable, the chain rule is used to compute the derivative of deeply nested functions. Suppose that $$y = f(g(x))$$ is a composite function, and that the underlying functions $$y = f(u)$$ and $$u = g(x)$$ are both differentiable. The chain rule states that the derivative of $$y$$ with respect to $$x$$ is the product of the derivative of $$y$$ with respect to $$u$$ and the derivative of $$u$$ with respect to $$x$$:

$$\frac{dy}{dx} = \frac{dy}{du} \frac{du}{dx}$$

Chain Rule for Single-Variable Functions

When a multivariate function outputs a vector rather than a scalar, the most natural representation of its derivative is the Jacobian matrix. Specifically, the derivative of a vector $$\mathbf{y}$$ with respect to an input vector $$\mathbf{x}$$ is a matrix that compiles the partial derivatives of each individual component of $$\mathbf{y}$$ with respect to each component of $$\mathbf{x}$$.

Jacobian Matrix

The partial derivative of a multivariate function $$y = f(x_1, x_2, \ldots, x_n)$$ with respect to its $$i^	extrm{th}$$ parameter $$x_i$$ measures how the function changes as $$x_i$$ increases while treating all other variables as constants. Formally, it is defined as the limit: $$\frac{\partial y}{\partial x_i} = \lim_{h ightarrow 0} \frac{f(x_1, \ldots, x_{i-1}, x_i+h, x_{i+1}, \ldots, x_n) - f(x_1, \ldots, x_i, \ldots, x_n)}{h}$$.

Partial Derivative

The gradient of a scalar-valued function with respect to a vector $$\mathbf{x}$$ is itself a vector-valued output that has the identical shape as the input vector $$\mathbf{x}$$. This property ensures that each element of the gradient vector directly corresponds to the partial derivative of the scalar output with respect to the matching element in the input vector.

Learn Before

Related