Gradient descent with momentum involves applying exponential smoothing to the computed gradient. This will speed up training, because the algorithm will oscillate less towards the minimum and it will take more steps towards the minimum. If exponential smoothing is unknown to you, you might want to read this article. Usually, simple exponential smoothing is used, meaning that there are two more hyperparameters to tune: the learning rate alpha and the smoothing parameter beta. Usually, this method almost always works better than traditional gradient descent, and it can be coupled with mini-batch gradient descent.