University of Michigan - Ann Arbor

Google

Human-Level Performance is the error a person may get. It's usually based on what your objective is. 
On the opposite to that is the Bayes (optimal) error which is the theoretical error. And this error may not be 0. Since there might be some impossible cases to classify, e.g., a picture that is too blurry to be recognized.
Before surpassing human-level performance, the accuracy will be improved greatly. However, when it passes the line, it will get harder since you cannot learn from human, and you don't know what the true Bayes error is.

Human Level Performance

In statistical classification, the Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error. In deep learning, human-level performance is often used as a proxy for the Bayes error rate.

Bayes Error Rate

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

To find a reasonable proxy for Bayes error, you should clearly identify what the purpose of your human level performance should be.
For example, since doctors have an error of 0.5%, in a given problem we would define 0.5% as the human level error as a proxy for Bayes error. 
The more specific in what you are defining, the better your outcome, e.g., the error rate by doctors who are above 50 years old and specialize in brain surgery.

Human Level Proxy for Bayes Error

You ask a few people to label the dataset to find out what is human-level performance. You find the following levels of accuracy:

- Expert #1: 0.3% error
- Expert #2: 0.5% error
- Normal Person #1 (not an expert): 1.0% error
- Normal Person #2 (not an expert): 1.2% error

If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”?

You define 0.1% as “human-level performance.” After working further on your algorithm, you end up with the following: 

- Human-level performance: 0.1%
- Training set error: 2.0%
- Dev set error: 2.1%

Human Level Performance: Based on the evidence below, which two of the following four options seem the most promising to try? 

Which of the following statements about algorithmic performance do you agree with?

The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

Bayes Classifier is the probabilistic model that makes the most probable prediction for new examples given the training data. Bayes error rate is the minimum possible error that can be made when making predictions.

Bayes Calssifier and Bayes Error Rate

Avoidable bias is the difference between a model's training error rate and the optimal error rate (Bayes error rate). In practice, since the Bayes error rate is difficult to calculate, human-level error is often used as a proxy. Avoidable bias is estimated as:
$$\text{Avoidable Bias} = \text{Training Error} - \text{Human-level Error}$$
A high avoidable bias indicates that the model is underfitting the training data.

Learn Before

Related