1. human error VS training set error: avoidable bias
2. Training set error VS training-dev set error: variance
3. training-dev set VS dev error: data mismatch
4. dev error VS test error: degree of overfitting to dev set

University of Michigan - Ann Arbor

You will face mismatch problem when your training set comes from a different distribution than your dev and test sets, e.g., training set from webpage images and dev/test sets from mobile app uploaded images.

Data Mismatch Problem in Deep Learning

Bias/variance on mismatched training and dev/test sets

1- Carry out manual error analysis to try to understand the difference between training and dev/test sets.

2- Make training data more similar or collect more data similar to dev/test sets.

3- Merge a subset of the dev/test set into the training set and use smaller dev/test sets. It's not a good idea to merge a subset of the training set into dev/test sets.

4- *Training-dev set*: a validation set with the same distribution as training set, but not used for training.

Learn Before

Related