https://towardsdatascience.com/the-3-best-optimization-methods-in-neural-networks-40879c887873

University of Michigan - Ann Arbor

Traditional gradient descent needs to process all of the training examples before making the first update to the parameters. From now, updating the parameters will be referred to as taking a step.

Now, we know that deep learning works best with large amounts of data. Therefore, gradient descent will need to train on millions of training points before taking a single step. This is obviously inefficient.


Instead, consider breaking up the test set into smaller sets. Each small set is called a mini-batch. Say each mini-batch has 64 training points. Then, we could train the algorithm on a mini-batch at a time and take a step once training is done for each mini-batch!

Mini-batch gradient descent


Optimization Methods in Neural Networks

if mini-batch size = m --> batch gradient descent: takes too much time, too long per iteration (blue line) 

if mini-batch size = 1 --> stochastic gradient descent: every example is its own mini-batch, lose speed up from vectorization (purple line) 

in practice: want to choose somewhere in between 1 & m for batch size --> fastest learning, vectorization, makes progress without processing entire training set (green line)

Learn Before

Related