When a model's parameters $$w$$ are fixed (not chosen to minimize training loss), the expected training error equals the expected test error. Both expectations are computed by drawing datasets from the same distribution $$p(x, y)$$, so they are mathematically identical. The discrepancy between training and test error arises only when parameters are selected to minimize training error on a particular sample, introducing an optimistic bias in training error relative to test error.

University of Michigan - Ann Arbor

Google

Typically, when training a machine learning model, we have access to a training set; we can compute some error measure on the training set, called the training error; and we reduce this training error. 

What separates machine learning from optimization is that we want the generalization error, also called the test error, to be low as well. The test error is deﬁned as the expected value of the error on a new input. Here the expectation is taken across diﬀerent possible inputs, drawn from the distribution of inputs we expect the system to encounter in practice. 

We typically estimate the test error of a machine learning model by measuring its performance on a test set of examples that were collected separately from the training set.

Training Error and Test Error

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

When we use a machine learning algorithm, we sample the training set, then use it to choose the parameters to reduce training set error, then sample the test set. Under this process, the expected test error is greater than or equal to the expected value of training error. 

The factors determining how well a machine learning algorithm will perform are its ability to:
1. Make the training error small.
2. Make the gap between training and test error small.

These two factors correspond to the two central challenges in machine learning: underﬁtting and overﬁtting. 


Factors determining how well a machine learning algorithm will perform

The generalization gap is the mathematical difference between a model's empirical training error ($$R_	extrm{emp}$$) and its expected generalization error ($$R$$), expressed as $$R_	extrm{emp} - R$$. If a model achieves a training error of exactly zero, the generalization gap becomes precisely equal to the generalization error, meaning that further predictive progress can only be made by reducing this gap.

Learn Before

Related