Avoidable bias:
(1) Train bigger model
(2) train longer/better optimization algorithms
(3) NN architecture/hyperparameters search

Variance:
(1) More data
(2) Regularization: L2, drop out, data augmentation...
(3) NN architecture/hyperparameters search

Improving deep learning model performance

- Train a bigger neural model.
- Train with better optimization algorithms.
- Find better neural network architecture.

Ways of Reducing Avoidable Bias

1. the difference between human error and training error: avoidable bias

2. the difference between training error and dev error: variations

University of Michigan - Ann Arbor

In statistical classification, Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error.

For deep learning, human-level performance can be regarded approximately as Bayes error rate.

Bayes error rate

- Human error VS training set error: avoidable bias
- Training set error VS training-dev set error: variance
- Training-dev set VS dev error: data mismatch
- Dev error VS test error: degree of overfitting to dev set

Possible Errors of a Supervised Learning model

The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

Human-Level Performance is the error a person may get. It's usually based on what your objective is. 
On the opposite to that is the Bayes (optimal) error which is the theoretical error. And this error may not be 0. Since there might be some impossible cases to classify, e.g., a picture that is too blurry to be recognized.
Before surpassing human-level performance, the accuracy will be improved greatly. However, when it passes the line, it will get harder since you cannot learn from human, and you don't know what the true Bayes error is.

Human Level Performance

Avoidable bias

You ask a few people to label the dataset to find out what is human-level performance. You find the following levels of accuracy:

- Expert #1: 0.3% error
- Expert #2: 0.5% error
- Normal Person #1 (not an expert): 1.0% error
- Normal Person #2 (not an expert): 1.2% error

If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”?

To find a reasonable proxy for Bayes error, you should clearly identify what the purpose of your human level performance should be.
For example, since doctors have an error of 0.5%, in a given problem we would define 0.5% as the human level error as a proxy for Bayes error. 
The more specific in what you are defining, the better your outcome, e.g., the error rate by doctors who are above 50 years old and specialize in brain surgery.

Human Level Proxy for Bayes Error

Which of the following statements about algorithmic performance do you agree with?

Bayes Classifier is the probabilistic model that makes the most probable prediction for new examples given the training data. Bayes error rate is the minimum possible error that can be made when making predictions.

Bayes Calssifier and Bayes Error Rate

- **High Avoidable bias**: if it does not fit training set well on cost function (e.g., comparable to human level performance), you might train a bigger network, train longer, switch to a better optimization algorithm, like the Adam, or change the neural network architecture/hyperparameter search.
- **High Variance**: if it does not fit dev set on cost function, you may use regularization, increase the size of training set, or change the neural network architecture/hyperparameter search.
- If it does not fit test set well on cost function, you may change or increase the size of dev set.
- If it does not perform well in real world (e.g., happy users), you may change or increase the size of test set or change the cost function.

Troubleshooting a deep learning model

You will face mismatch problem when your training set comes from a different distribution than your dev and test sets, e.g., training set from webpage images and dev/test sets from mobile app uploaded images.

Learn Before

Related

Learn After