The left plot shows fitting a straight line to the data. This is not a very good fit. So, it is underfitting the data and we call it high bias. On the opposite end, if you fit an incredibly complex classifier, maybe you can fit the data perfectly, but that doesn't look like a great fit either. So there's a classifier of high variance and this is overfitting the data. And there might be some classifier in between, with a medium level of complexity, that maybe fits it correctly.

Overfitting/Underfitting vs. Bias/Variance in Supervised Machine Learning

After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


Which of the following would be the best choice for the next ridge regression model you train?

1. Cross-validation
2. Train with more data.
3. Remove features.
4. Early stopping.
5. Regularization.
6. Ensembling.

How to avoid overfitting

While guarantees from classical learning theory (such as bounds based on the Vapnik-Chervonenkis (VC) dimension or Rademacher complexity) can be conservative even for classical models, they appear powerless to explain why deep neural networks generalize. For classification problems, deep models are typically expressive enough to perfectly fit arbitrary labels even for datasets consisting of millions of examples. In the classical picture, such extreme model complexity—even when utilizing familiar methods like $$\ell_2$$ regularization—should lead to severe overfitting. Paradoxically, despite perfectly fitting the training data with zero training error, these highly expressive models often generalize remarkably well to unseen data, contradicting traditional complexity-based generalization bounds.

Generalization Paradox in Deep Learning

A common method to determine whether a supervised statistical model is overfitting is to compare its performance on the training data against unseen test data. If the model achieves very high accuracy on the training data but significantly lower accuracy on the testing data, it is likely overfitting.

Identifying Overfitting via Training and Test Performance

Increasing the flexibility of a supervised statistical model can lead to overfitting, where the trained model $$\hat{f}$$ fits the noise or random errors in the training observations rather than approximating the true underlying function $$f$$. Consequently, when selecting a model class and training a supervised statistical model, one must balance model flexibility with the risk of overfitting. Models that are too complex relative to the size of the training dataset are highly susceptible to overfitting and typically fail to generalize well to new, unseen examples.

University of Michigan - Ann Arbor

Claude

Google

A model’s capacity is its ability to fit a wide variety of functions. The model $$\hat{f}$$ that we estimate by choosing a functional form and training, is just an approximation and almost never matches the true unknown $$f$$. To have a closer estimate, we can choose more flexible functional forms with more parameters to estimate. Machine learning algorithms will generally perform best when their capacity is appropriate for the true complexity of the task they need to perform and the amount of training data they are provided with. Models with insufficient capacity are unable to solve complex tasks. Models with high capacity can solve complex tasks, but when their capacity is higher than needed to solve the present task, they may overfit.

Supervised Statistical Model Flexibility (Capacity/Complexity)

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Dive into Deep Learning

Overfitting a supervised statistical model

Typically, when training a machine learning model, we have access to a training set; we can compute some error measure on the training set, called the training error; and we reduce this training error. 

What separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. The test error is deﬁned as the expected value of the error on a new input. Here the expectation is taken across diﬀerent possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice. 

We typically estimate the test error of a machine learning model by measuring its performance on a test set of examples that were collected separately from the training set.

Training Error and Test Error

Generalizability refers to an algorithm's ability to give accurate predictions for new, previously unseen data.

Generalizability of a supervised statistical model

Models that are too simple that don't even do well on the training data, are said to **underfit** and are not likely to generalize well to new examples.

Underfitting a supervised statistical model

Rademacher complexity provides a method of quantifying a model's complexity to ensure that a training set is representative of the population. This is done by calculating the probability distribution of the training set from the sample and the potential classifiers with their related error functions.

Measuring Model Complexity: Rademacher complexity

Bias is an error that occurs when a model is unable to accurately represent the true $$f$$ as a result of underlying assumptions made to simplify the model. For example, using linear regression for a relationship that is not linear in nature has a large amount of bias since no linear estimate will result in a model that accurately captures the non-linear nature of $$f$$. Its flexibility is limited and often underfits the data. More flexible methods result in less bias since they are able to more accurately represent the true $$f$$.

Bias of Supervised Models in Statistical Learning

Variance refers to the sensitivity of a model ($ \hat{f} $) to different training data sets. In other words, it is the difference in the fits of $ \hat{f} $ between different training data sets. When we use data sets to fit our statistical learning method, each one will produce a different $ \hat{f} $; we want an $ \hat{f} $ that varies very little between them. High variance means $ \hat{f} $ is often more flexible and captures the points of that particular data set well, but changing points for a different training data set will result in large changes in $ \hat{f} $ due to overfitting. Lower variance results in less flexibility, but does not change $ \hat{f} $ as much between different training data sets.

Variance of Supervised Models in Statistical Learning

In machine learning, model complexity can be analyzed through Karl Popper's scientific criterion of falsifiability. An overly complex model class that can perfectly fit any arbitrary dataset—including randomly assigned labels—acts like an unfalsifiable theory that explains every possible observation, failing to guarantee that a true pattern has been discovered. Conversely, if a model class is restrictive enough that it could not possibly fit arbitrary labels, yet it successfully fits the actual training data, we can more confidently conclude it has learned a generalizable pattern.

Falsifiability of Machine Learning Models

Determining an appropriate measure for model complexity is nuanced and varies across different classes of machine learning models. While complexity is frequently associated with the number of parameters—since models with more parameters can typically fit more arbitrarily assigned labels—this is not always true. For example, kernel methods can operate in spaces with an infinite number of parameters, yet their complexity is controlled through other mechanisms. A more useful notion of complexity often involves the range of values that parameters can take; a model whose parameters can take arbitrary values is considered more complex. Consequently, comparing complexity between substantially different model classes, such as decision trees and neural networks, can be difficult.

Notions of Model Complexity

The amount of available training data dictates the appropriate level of model complexity. For a fixed task and data distribution, model complexity should not increase more rapidly than the dataset size. With fewer training samples, a model is highly susceptible to overfitting, making simpler models difficult to beat. However, as dataset size increases, generalization error typically decreases, allowing for the successful training of more complex architectures like deep neural networks.

Learn Before

Related

Learn After