University of Michigan - Ann Arbor

In statistical classification, Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error.

For deep learning, human-level performance can be regarded approximately as Bayes error rate.

Bayes error rate

Human-Level Performance is the error a person may get. It's usually based on what your objective is. 
On the opposite to that is the Bayes (optimal) error which is the theoretical error. And this error may not be 0. Since there might be some impossible cases to classify, e.g., a picture that is too blurry to be recognized.
Before surpassing human-level performance, the accuracy will be improved greatly. However, when it passes the line, it will get harder since you cannot learn from human, and you don't know what the true Bayes error is.

Human Level Performance

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

1. the difference between human error and training error: avoidable bias

2. the difference between training error and dev error: variations

Avoidable bias

You ask a few people to label the dataset to find out what is human-level performance. You find the following levels of accuracy:

- Expert #1: 0.3% error
- Expert #2: 0.5% error
- Normal Person #1 (not an expert): 1.0% error
- Normal Person #2 (not an expert): 1.2% error

If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”?

To find a reasonable proxy for Bayes error, you should clearly identify what the purpose of your human level performance should be.
For example, since doctors have an error of 0.5%, in a given problem we would define 0.5% as the human level error as a proxy for Bayes error. 
The more specific in what you are defining, the better your outcome, e.g., the error rate by doctors who are above 50 years old and specialize in brain surgery.

Human Level Proxy for Bayes Error

Which of the following statements about algorithmic performance do you agree with?

Bayes Classifier is the probabilistic model that makes the most probable prediction for new examples given the training data. Bayes error rate is the minimum possible error that can be made when making predictions.

Bayes Calssifier and Bayes Error Rate

You define 0.1% as “human-level performance.” After working further on your algorithm, you end up with the following: 

- Human-level performance: 0.1%
- Training set error: 2.0%
- Dev set error: 2.1%

Learn Before

Related