1. human error VS training set error: avoidable bias
2. Training set error VS training-dev set error: variance
3. training-dev set VS dev error: data mismatch
4. dev error VS test error: degree of overfitting to dev set

Bias/variance on mismatched training and dev/test sets

To address a mismatch between training data and development (dev) or test data: (1) perform manual error analysis to identify differences between the distributions; (2) collect or create training examples that better resemble the dev/test distribution; and (3) if appropriate, move a subset of dev/test examples into training and rebuild smaller dev and test sets, rather than moving training examples into the evaluation sets. A training-dev set is held out from training but sampled from the training distribution, allowing its performance to be compared with dev-set performance when diagnosing mismatch.

Strategies for Addressing Training–Dev/Test Data Mismatch

You will face mismatch problem when your training set comes from a different distribution than your dev and test sets, e.g., training set from webpage images and dev/test sets from mobile app uploaded images.

University of Michigan - Ann Arbor

- Human error VS training set error: avoidable bias
- Training set error VS training-dev set error: variance
- Training-dev set VS dev error: data mismatch
- Dev error VS test error: degree of overfitting to dev set

Possible Errors of a Supervised Learning model

- **High Avoidable bias**: if it does not fit training set well on cost function (e.g., comparable to human level performance), you might train a bigger network, train longer, switch to a better optimization algorithm, like the Adam, or change the neural network architecture/hyperparameter search.
- **High Variance**: if it does not fit dev set on cost function, you may use regularization, increase the size of training set, or change the neural network architecture/hyperparameter search.
- If it does not fit test set well on cost function, you may change or increase the size of dev set.
- If it does not perform well in real world (e.g., happy users), you may change or increase the size of test set or change the cost function.

Troubleshooting a deep learning model

Avoidable bias is the difference between a model's training error rate and the optimal error rate (Bayes error rate). In practice, since the Bayes error rate is difficult to calculate, human-level error is often used as a proxy. Avoidable bias is estimated as:
$$\text{Avoidable Bias} = \text{Training Error} - \text{Human-level Error}$$
A high avoidable bias indicates that the model is underfitting the training data.

Avoidable bias

A model with 10% training error, 11% training-dev error, and 12% development error has only a one-percentage-point increase at each comparison. The small training-to-training-dev gap suggests low variance on the training distribution, while the small training-dev-to-development gap suggests little data mismatch. Whether the 10% training error represents high avoidable bias depends on the best achievable, Bayes, or human-level error: high avoidable bias is supported only if that benchmark is substantially lower than 10%.

Interpreting 10% Training, 11% Training-Dev, and 12% Development Error

Suppose the best possible error is near 0%, while a defect-detection model has 1% training error, 5% training-dev error, and 5% dev error. The four-percentage-point increase from training to training-dev error indicates high variance because the model performs substantially worse on new examples from the training distribution. The 1% training error suggests little avoidable bias, and the equal training-dev and dev errors suggest little data mismatch.

Learn Before

Related

Learn After

Learn After

Related