The minimum of the empirical risk function on a training dataset can occur at a completely different location than the minimum of the true risk function. This illustrates why simply minimizing the training error via optimization may not necessarily lead to the lowest possible generalization error.

Minima of Empirical Risk and Risk

Empirical risk is defined as the empirical average of the loss function. In this case, empirical means the performance was evaluated with real-life testing.

University of Virginia, Charlottesville

University of Michigan - Ann Arbor

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

The best strategy to improve optimization is by designing models that are easier to optimize. For example, linear paths or skip connections between layers reduce the length of the shortest path from the lower layer’s parameters to the output, and thus mitigate the vanishing gradient problem. Model innovations like the LSTM, rectiﬁed linear units and maxout units have all moved toward using linear functions to improve optimization. 

Designing Models to Aid Optimization

Machine learning acts indirectly, specifically, with loss fucntion having relationship with but not the same with the measurement P, which pure optimization always wants to optimize.

How does learning differ from pure optimization?

The training process based on minimizing this average training error is known
as empirical risk minimization.

Empirical Risk Minimization

This method of optimization involves applying algorithm concepts to training other algorithms, hence the "meta-" prefix. 

Meta-Algorithm Approaches

Empirical Risk

Surrogate loss function is a proxy having consistency or calibration with the original loss function, which is not comfortable for people to use derivatives or other direct method to optimize.

Surrogate Loss Function

Early Stopping means stopping earlier when the loss function doesn't have relatively obvious progress or with worse validation in one or several recent epochs.

Early Stopping

The underlying premise:
- In high dimensional feature spaces, there is a very large likelihood of encountering multiple local minima. 
- This is compounded with the fact that the feature space isn’t well behaved i.e. an inevitable presence of plateaus, saddle points affecting the magnitude of the gradient along with the direction of the gradient not pointing towards the approximate region of lower cost.

Thus, being able to initialize parameters in a region smaller than the feature space itself and one that behaves well enough for the descent to lead to a reasonable solution is powerful. This is exactly what continuation methods try to achieve.

Continuation Methods

Indexed notation is commonly used to represent a subsequence of elements, such as vectors. A sequence starting from index $m$ and ending at index $len$ is written as $x_m, x_{m+1}, \dots, x_{len-1}, x_{len}$. In the context of sequence models, this notation is frequently used for different components. For instance, a sequence of hidden states is often denoted as $h_m, h_{m+1}, \dots, h_{len}$, while a sequence of embeddings might be represented as $\mathbf{e}_m, \mathbf{e}_{m+1}, \dots, \mathbf{e}_{len}$.

Indexed Sequence Notation

The primary goal of optimization in the context of deep learning is to minimize an objective function. Since this objective function is usually a loss function calculated from the training dataset, optimization algorithms are primarily concerned with reducing the training error, also known as empirical risk.

Learn Before

Related

Learn After