A linear function ﬁt to the data suﬀers from underﬁtting—it cannot capture the curvature that is present in the data.

Aquadratic function ﬁt to the data generalizes well to unseen points. It does not suﬀer froma signiﬁcant amount of overﬁtting or underﬁtting.

A polynomial of degree 9 ﬁt to the data suﬀers from overﬁtting. The solution passes through all the training points exactly, but we have not been lucky enough for it to extract the correct structure.


University of Michigan - Ann Arbor

When we use a machine learning algorithm, we sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. Under this process, the expected test error is greater than or equal to the expected value of training error. 

The factors determining how well a machine learning algorithm will perform are its ability to:
1. Make the training error small.
2. Make the gap between training and test error small.

These two factors correspond to the two central challenges in machine learning: underﬁtting and overﬁtting. 


Factors determining how well a machine learning algorithm will perform

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

An Example for Underfitting vs. Appropriate Capacity vs. Overfitting

Below is a representation of under and overfitting of the boundaries of U.S. sates. The data comes from the US Census Bureau. In its original format, the data is a single Keyhole Markup Language (KML) file which contains latitude and longitude coordinates of the borders of US states. The necessary latitude, longitude, and label (state) data were parsed from the KML files using a simple Python script. The main idea here is to understand the bias-variance trade-off and how that relates to under and overfitting. 

Additionally there are two examples of ways they avoided under and overfitting and created a much more accurate map using (gradient) boosting and random forest classifiers.

Learn Before

Related