On iteration t:
         Compute dW, db on the current mini-batch
                $v_{dW} = \beta v_{dW} + (1-\beta)dW$
                $v_{db} = \beta v_{db} + (1-\beta)db$
                $W = W - \alpha v_{dW}, b = b - \alpha v_{db}$
Note that now we have two parameters $\alpha$ and $\beta$. 

University of Michigan - Ann Arbor

The basic idea of gradient descent with momentum is to compute an exponentially weighted average of your gradients, and then use that gradient to update your weights instead. It almost always works faster than the standard gradient descent algorithm.


Gradient Descent with Momentum

As shown in the picture below, for Gradient descent optimizer, we will have ups and downs in the vertical direction, but it continues to go right in the horizontal direction. By taking the average of the few previous gradients, you will decrease oscillations in the vertical direction by averaging out positive and negative values. And since all gradients point to the same direction horizontally, the result in the horizontal direction will remain a large value in the right direction.

Intuition behind Gradient Descent with Momentum

These plots were generated with gradient descent; with gradient descent with momentum (β = 0.5) and gradient descent with momentum (β = 0.9). Which curve corresponds to which algorithm?

Gradient Descent with Momentum Pseudocode 

Adam stands for adaptive moment estimation.
It combines gradient descent with momentum, and RMSProp. It brings the benefits from both sides - adaptive learning rate and faster convergence with momentum.

Learn Before

Related