https://elitedatascience.com/overfitting-in-machine-learning
Overfitting in Machine Learning: What It Is and How to Prevent It

Overfitting in Machine Learning: What It Is and How to Prevent It

1. Cross-validation
2. Train with more data.
3. Remove features.
4. Early stopping.
5. Regularization.
6. Ensembling.

University of Michigan - Ann Arbor

Increasing the flexibility of a supervised statistical model may result in overfitting, such that the trained model $\hat{f}$ gets prone to the noise or errors in the observations (training data) rather than approximating the true $f$.
So, when choosing a functional form and training a supervised statistical model, we should always consider a trade-off between the flexibility and overfitting of $\hat{f}$.

Models that re too complex for the amount of training data available are said to **overfit** and are not likely to generalize well to new examples.

Overfitting a supervised statistical model

One common way to detirmine whether or not your model is overfitting is by analyzing the accuracy of the model's performance on the training data and test data. Your model may be overefit if you have very high accuracy on the training data, and then much lower accuracy on the testing data.

How to tell when your model is overfitting

The left plot shows fitting a straight line to the data. This is not a very good fit. So, it is underfitting the data and we call it high bias. On the opposite end, if you fit an incredibly complex classifier, maybe you can fit the data perfectly, but that doesn't look like a great fit either. So there's a classifier of high variance and this is overfitting the data. And there might be some classifier in between, with a medium level of complexity, that maybe fits it correctly.

Overfitting/Underfitting vs. Bias/Variance in Supervised Machine Learning

After training a ridge regression model, you find that the training and test set accuracies are 0.98 and 0.54 respectively.


Which of the following would be the best choice for the next ridge regression model you train?

How to avoid overfitting

While guarantees from classical learning theory (such as bounds based on the Vapnik-Chervonenkis (VC) dimension or Rademacher complexity) can be conservative even for classical models, they appear powerless to explain why deep neural networks generalize. For classification problems, deep models are typically expressive enough to perfectly fit arbitrary labels even for datasets consisting of millions of examples. In the classical picture, such extreme model complexity—even when utilizing familiar methods like $$\ell_2$$ regularization—should lead to severe overfitting. Paradoxically, despite perfectly fitting the training data with zero training error, these highly expressive models often generalize remarkably well to unseen data, contradicting traditional complexity-based generalization bounds.

Learn Before

Related

Learn After