In deep learning, optimization serves purposes beyond simply minimizing the empirical training error. The specific choice of optimization algorithms and learning rate schedules can lead to significantly different levels of generalization and overfitting on the test set, even when those differing choices result in the exact same amount of training error.