Traditional gradient descent needs to process all of the training examples before making the first update to the parameters. From now, updating the parameters will be referred to as taking a step.

Now, we know that deep learning works best with large amounts of data. Therefore, gradient descent will need to train on millions of training points before taking a single step. This is obviously inefficient.

Instead, consider breaking up the test set into smaller sets. Each small set is called a mini-batch. Say each mini-batch has 64 training points. Then, we could train the algorithm on a mini-batch at a time and take a step once training is done for each mini-batch!