A common method to determine whether a supervised statistical model is overfitting is to compare its performance on the training data against unseen test data. If the model achieves very high accuracy on the training data but significantly lower accuracy on the testing data, it is likely overfitting.

University of Michigan - Ann Arbor

Google

Increasing the flexibility of a supervised statistical model can lead to overfitting, where the trained model $$\hat{f}$$ fits the noise or random errors in the training observations rather than approximating the true underlying function $$f$$. Consequently, when selecting a model class and training a supervised statistical model, one must balance model flexibility with the risk of overfitting. Models that are too complex relative to the size of the training dataset are highly susceptible to overfitting and typically fail to generalize well to new, unseen examples.

Overfitting a supervised statistical model

The left plot shows fitting a straight line to the data. This is not a very good fit. So, it is underfitting the data and we call it high bias. On the opposite end, if you fit an incredibly complex classifier, maybe you can fit the data perfectly, but that doesn't look like a great fit either. So there's a classifier of high variance and this is overfitting the data. And there might be some classifier in between, with a medium level of complexity, that maybe fits it correctly.

Overfitting/Underfitting vs. Bias/Variance in Supervised Machine Learning

After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


Which of the following would be the best choice for the next ridge regression model you train?

1. Cross-validation
2. Train with more data.
3. Remove features.
4. Early stopping.
5. Regularization.
6. Ensembling.

How to avoid overfitting

While guarantees from classical learning theory (such as bounds based on the Vapnik-Chervonenkis (VC) dimension or Rademacher complexity) can be conservative even for classical models, they appear powerless to explain why deep neural networks generalize. For classification problems, deep models are typically expressive enough to perfectly fit arbitrary labels even for datasets consisting of millions of examples. In the classical picture, such extreme model complexity—even when utilizing familiar methods like $$\ell_2$$ regularization—should lead to severe overfitting. Paradoxically, despite perfectly fitting the training data with zero training error, these highly expressive models often generalize remarkably well to unseen data, contradicting traditional complexity-based generalization bounds.

Learn Before

Related