Underfitting occurs when a model is insufficiently expressive to capture the underlying patterns in the training data. This situation is identified when both the training error and validation error are substantial, but the generalization gap ($$R_	extrm{emp} - R$$) between them remains small. Because the model fails to reduce the training error despite a small gap, overall predictive performance could likely be improved by utilizing a more complex model.

Underfitting

Overfitting occurs when a model performs significantly better on its training data than on validation or holdout data, resulting in a large generalization gap. This indicates that the model is overly complex and has begun to memorize noise rather than learning generalizable patterns. However, severe overfitting is not always inherently detrimental in deep learning; the best predictive models often exhibit this behavior. The primary goal is to drive the overall generalization error lower, meaning the gap is only problematic if it obstructs that goal.

Overfitting

Uniform convergence is a theoretical principle providing a guarantee that, with high probability, the empirical error rate for every classifier $$f$$ within a predefined model class $$\mathcal{F}$$ will simultaneously converge to its true population error rate. Specifically, it ensures that with a probability of at least $$1-\delta$$, no classifier's error rate in the class will be misestimated by more than a small margin $$\alpha$$.

Uniform Convergence in Machine Learning

The generalization gap is the mathematical difference between a model's empirical training error ($$R_	extrm{emp}$$) and its expected generalization error ($$R$$), expressed as $$R_	extrm{emp} - R$$. If a model achieves a training error of exactly zero, the generalization gap becomes precisely equal to the generalization error, meaning that further predictive progress can only be made by reducing this gap.

Claude

Typically, when training a machine learning model, we have access to a training set; we can compute some error measure on the training set, called the training error; and we reduce this training error. 

What separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. The test error is deﬁned as the expected value of the error on a new input. Here the expectation is taken across diﬀerent possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice. 

We typically estimate the test error of a machine learning model by measuring its performance on a test set of examples that were collected separately from the training set.

Training Error and Test Error

Dive into Deep Learning

One immediate connection we can observe between training error and test error is that the expected training error of a randomly selected model is equal to the expected test error of that model. 

Suppose we have a probability distribution $p(x, y)$ and we sample from it repeatedly to generate the training set and the test set. For some ﬁxed value $w$, the expected training set error is exactly the same as the expected test set error, because both expectations are formed using the same dataset sampling process. The only diﬀerence between the two conditions is the name we assign to the dataset we sample.


Connection between Training Error and Test Error

When we use a machine learning algorithm, we sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. Under this process, the expected test error is greater than or equal to the expected value of training error. 

The factors determining how well a machine learning algorithm will perform are its ability to:
1. Make the training error small.
2. Make the gap between training and test error small.

These two factors correspond to the two central challenges in machine learning: underﬁtting and overﬁtting. 


Learn Before

Related

Learn After