Batch size controls the accuracy of the estimate of the error gradient when training neural networks, and training tends to become more stable as it progresses. Smaller batch sizes are commonly preferred for two main reasons: (1) they introduce noise into the gradient estimate, which has a regularizing effect and can lower generalization error, and (2) a smaller batch is easier to fit in memory during training (e.g., on a GPU). In practice, batch size is often set to a small value such as 32 and is not treated as a hyperparameter to be tuned.