1. human error VS training set error: avoidable bias
2. Training set error VS training-dev set error: variance
3. training-dev set VS dev error: data mismatch
4. dev error VS test error: degree of overfitting to dev set

Bias/variance on mismatched training and dev/test sets

1- Carry out manual error analysis to try to understand the difference between training and dev/test sets.

2- Make training data more similar or collect more data similar to dev/test sets.

3- Merge a subset of the dev/test set into the training set and use smaller dev/test sets. It's not a good idea to merge a subset of the training set into dev/test sets.

4- *Training-dev set*: a validation set with the same distribution as training set, but not used for training.

Solving Data Mismatch Problem in Deep Learning

You will face mismatch problem when your training set comes from a different distribution than your dev and test sets, e.g., training set from webpage images and dev/test sets from mobile app uploaded images.

University of Michigan - Ann Arbor

- Human error VS training set error: avoidable bias
- Training set error VS training-dev set error: variance
- Training-dev set VS dev error: data mismatch
- Dev error VS test error: degree of overfitting to dev set

Possible Errors of a Supervised Learning model

- **High Avoidable bias**: if it does not fit training set well on cost function (e.g., comparable to human level performance), you might train a bigger network, train longer, switch to a better optimization algorithm, like the Adam, or change the neural network architecture/hyperparameter search.
- **High Variance**: if it does not fit dev set on cost function, you may use regularization, increase the size of training set, or change the neural network architecture/hyperparameter search.
- If it does not fit test set well on cost function, you may change or increase the size of dev set.
- If it does not perform well in real world (e.g., happy users), you may change or increase the size of test set or change the cost function.

Troubleshooting a deep learning model

Data Mismatch Problem in Deep Learning

1. the difference between human error and training error: avoidable bias

2. the difference between training error and dev error: variations

Learn Before

Related

Learn After