A defining characteristic of deep learning research and practice is its strong reliance on empiricism, particularly when dealing with complex, nonconvex nonlinear optimization problems. Instead of requiring formal mathematical proofs of optimality or convergence before adopting an algorithm, practitioners prioritize rapid experimentation. They are willing to accept theoretically suboptimal solutions if those solutions yield superior practical accuracy, an approach that has significantly accelerated the development of practical deep learning tools.

Empiricism in Deep Learning

Once a data representation, a model, and a well-defined objective function are established, an optimization algorithm is required. This computational algorithm searches for the best possible model parameters to effectively minimize the loss function on the dataset.

Claude

A loss function is a specific type of objective function where the convention is that lower values indicate better model performance. Optimization algorithms actively seek to minimize the loss function to improve the model; any objective where higher is better can be converted into a loss function by simply flipping its sign.

Loss Function

Dive into Deep Learning

Machine Learning Optimization Algorithm

The squared error is the standard loss function for regression problems, used to quantify the discrepancy between a single predicted value $$\hat{y}^{(i)}$$ and its true corresponding label $$y^{(i)}$$. The loss for a single example is defined mathematically as $$l^{(i)}(\mathbf{w}, b) = \frac{1}{2} \left(\hat{y}^{(i)} - y^{(i)}ight)^2$$. The quadratic form heavily penalizes large differences, and the constant fraction $$\frac{1}{2}$$ is conventionally added because it cleanly cancels out when taking the derivative during optimization.

Squared Error Loss

Classical regularization techniques mitigate overfitting by adding a parameter norm penalty to the loss function during training. This extra penalty term restricts the capacity of the model, thereby keeping the learned model simple and reducing its overall complexity. Different parameter norm penalties lead to different regularization behaviors and preferred solutions.

Learn Before

Related

Learn After