In deep learning, optimization serves purposes beyond simply minimizing the empirical training error. The specific choice of optimization algorithms and learning rate schedules can lead to significantly different levels of generalization and overfitting on the test set, even when those differing choices result in the exact same amount of training error.

Impact of Optimization Choices on Generalization

The goals of optimization and deep learning are fundamentally different despite being closely related. Optimization focuses on minimizing an objective function based on a training dataset to reduce training error, whereas deep learning aims to discover a model that generalizes well to the entire population of data to reduce generalization error (risk).

Claude

The primary goal of optimization in the context of deep learning is to minimize an objective function. Since this objective function is usually a loss function calculated from the training dataset, optimization algorithms are primarily concerned with reducing the training error, also known as empirical risk.

Learn Before

Related

Learn After