Early Stopping is designed to protect model with overfitting.

Why we need early stopping?

A machine learning model is being trained, and its performance is measured at the end of each epoch. The table below shows the training loss (how well the model fits the data it was trained on) and the validation loss (how well the model performs on a separate set of unseen data). Based on these results, after which epoch should the training process be stopped to ensure the model generalizes best to new, unseen data? Justify your decision.

Model Training Performance Analysis

A machine learning engineer is training a large model and observes the following: over the last 10 training cycles, the error on the training dataset has continued to decrease, but the error on a separate validation dataset has consistently increased. What is the most appropriate immediate action to take?

When training a predictive model, a decision must be made about when to stop the training process. Describe the primary risk associated with stopping the training process too soon, and the primary risk associated with continuing the training process for too long. What is the ultimate objective that guides the decision of when to stop?

The Trade-off in Model Training Duration

Early Stopping means stopping earlier when the loss function doesn't have relatively obvious progress or with worse validation in one or several recent epochs.

University of Michigan - Ann Arbor

Google

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Reference of Foundations of Large Language Models Course

The best strategy to improve optimization is by designing models that are easier to optimize. For example, linear paths or skip connections between layers reduce the length of the shortest path from the lower layer’s parameters to the output, and thus mitigate the vanishing gradient problem. Model innovations like the LSTM, rectiﬁed linear units and maxout units have all moved toward using linear functions to improve optimization. 

Designing Models to Aid Optimization

Machine learning acts indirectly, specifically, with loss fucntion having relationship with but not the same with the measurement P, which pure optimization always wants to optimize.

How does learning differ from pure optimization?

The training process based on minimizing this average training error is known
as empirical risk minimization.

Empirical Risk Minimization

This method of optimization involves applying algorithm concepts to training other algorithms, hence the "meta-" prefix. 

Meta-Algorithm Approaches

Empirical risk is defined as the empirical average of the loss function. In this case, empirical means the performance was evaluated with real-life testing.

Empirical Risk

Surrogate loss function is a proxy having consistency or calibration with the original loss function, which is not comfortable for people to use derivatives or other direct method to optimize.

Surrogate Loss Function

Early Stopping

The underlying premise:
- In high dimensional feature spaces, there is a very large likelihood of encountering multiple local minima. 
- This is compounded with the fact that the feature space isn’t well behaved i.e. an inevitable presence of plateaus, saddle points affecting the magnitude of the gradient along with the direction of the gradient not pointing towards the approximate region of lower cost.

Thus, being able to initialize parameters in a region smaller than the feature space itself and one that behaves well enough for the descent to lead to a reasonable solution is powerful. This is exactly what continuation methods try to achieve.

Continuation Methods

Indexed notation is commonly used to represent a subsequence of elements, such as vectors. A sequence starting from index $m$ and ending at index $len$ is written as $x_m, x_{m+1}, \dots, x_{len-1}, x_{len}$. In the context of sequence models, this notation is frequently used for different components. For instance, a sequence of hidden states is often denoted as $h_m, h_{m+1}, \dots, h_{len}$, while a sequence of embeddings might be represented as $\mathbf{e}_m, \mathbf{e}_{m+1}, \dots, \mathbf{e}_{len}$.

Learn Before

Related

Learn After