As shown in the picture below, for Gradient descent optimizer, we will have ups and downs in the vertical direction, but it continues to go right in the horizontal direction. By taking the average of the few previous gradients, you will decrease oscillations in the vertical direction by averaging out positive and negative values. And since all gradients point to the same direction horizontally, the result in the horizontal direction will remain a large value in the right direction.

University of Michigan - Ann Arbor

The basic idea of gradient descent with momentum is to compute an exponentially weighted average of your gradients, and then use that gradient to update your weights instead. It almost always works faster than the standard gradient descent algorithm.


Gradient Descent with Momentum

https://www.coursera.org/learn/deep-neural-network?specialization=deep-learning

Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization

A helpful website for understanding gradient descent:
https://towardsdatascience.com/gradient-descent-explained-9b953fc0d2c

Gradient Descent Reference

Intuition behind Gradient Descent with Momentum

These plots were generated with gradient descent; with gradient descent with momentum (β = 0.5) and gradient descent with momentum (β = 0.9). Which curve corresponds to which algorithm?

On iteration t:
         Compute dW, db on the current mini-batch
                $v_{dW} = \beta v_{dW} + (1-\beta)dW$
                $v_{db} = \beta v_{db} + (1-\beta)db$
                $W = W - \alpha v_{dW}, b = b - \alpha v_{db}$
Note that now we have two parameters $\alpha$ and $\beta$. 

Gradient Descent with Momentum Pseudocode 

Adam stands for adaptive moment estimation.
It combines gradient descent with momentum, and RMSProp. It brings the benefits from both sides - adaptive learning rate and faster convergence with momentum.

Learn Before

Related