Standard momentum calculates the gradient at the current parameter position and then takes a step based on both the gradient and accumulated momentum. If the momentum carries the parameters past a minimum, the gradient from the original position may cause the algorithm to overshoot or diverge.

Nesterov momentum improves upon this by first taking a "lookahead" step in the direction of the accumulated momentum, and then calculating the gradient at this new position. By evaluating the gradient after the momentum step, Nesterov momentum can apply a corrective update. If the momentum step overshoots the minimum, the gradient at the lookahead position will point back toward the minimum, allowing the algorithm to adjust its course and converge more stably.

University of California, Berkeley

Google

In the momentum method we basically first moved our weight in the direction of the current gradient and then moved in the direction  of momentum (weighted sum of all previous steps). Now in the new method we first move in the direction of the momentum and then calculate the gradient at the new point. Using this gradient we move in the direction of the new gradient. 

The update rules are as follows:
$$v \leftarrow \alpha v - \epsilon \nabla_{\theta} [\frac{1}{m} \sum^{m}_{i=1} L(f(x^{(i)};\theta + \alpha v), y^{(i)})]$$
$$\theta \leftarrow \theta + v$$

Nesterov momentum (Deep Learning Optimization Algorithm)

Pros:
- In practice Nesterov Momentum works better than Momentum Method

Cons:
- Both of the momentum algorithms are very sensitive to the choice of the learning rate

Pros and Cons of Nesterov Momentum

The formula for the Nesterov algorithm is:

$$step_{t} = \beta step_{t-1} + \alpha \nabla J(W^{t-1} - \beta step_{t-1})$$

$$W^{t} = W^{t-1} - step_{t}$$

Where:
- $$\nabla J(W^{t-1} - \beta step_{t-1})$$ is the gradient calculated for the new point where $$J$$ is the cost function.
- $$step_{t}$$ is the step at time stamp $$t$$.
- $$W^{t}$$ are the parameters for the layer at the time stamp $$t$$.
- $$\alpha$$ is the learning rate.
- $$\beta$$ is another hyperparameter (mostly people use 0.9).

Also the same process is done for bias parameters.

Learn Before

Related