The test error rate of the Bayes classifier on a set of data is known as the Bayes error rate. The Bayes error rate is the minimum possible test error rate for any classifier on that set of data. 

Formula for Bayes error rate:

$$1 - E(max_{j} Pr(Y = j|X))$$

Derivation:
Given an observation with predictor vector $$x_o$$, the Bayes classifier always assigns class $$j$$ to the observation where $$j$$ is the most likely class of the observation given $$x_o$$. Formally, it picks $$j$$ for which $$Pr(Y = j | X = x_o)$$  is maximal. It then follows that the error rate for predicting an observation with predictor vector $$x_o$$ is simply the probability that the observation belongs to a class other than the most likely given $$x_o$$. Formally, this probability is $$1 -  max_jPr(Y = j | X = x_o)$$. Finally, this means that the Bayes classifier has an error rate of $$1 -  E(max_jPr(Y = j | X ))$$, which is the average of the error rates for every possible of X.

Bayes Error Rate for (Naive) Bayes Classifier

Human-Level Performance is the error a person may get. It's usually based on what your objective is. 
On the opposite to that is the Bayes (optimal) error which is the theoretical error. And this error may not be 0. Since there might be some impossible cases to classify, e.g., a picture that is too blurry to be recognized.
Before surpassing human-level performance, the accuracy will be improved greatly. However, when it passes the line, it will get harder since you cannot learn from human, and you don't know what the true Bayes error is.

Human Level Performance

1. the difference between human error and training error: avoidable bias

2. the difference between training error and dev error: variations

Avoidable bias

You ask a few people to label the dataset to find out what is human-level performance. You find the following levels of accuracy:

- Expert #1: 0.3% error
- Expert #2: 0.5% error
- Normal Person #1 (not an expert): 1.0% error
- Normal Person #2 (not an expert): 1.2% error

If your goal is to have “human-level performance” be a proxy (or estimate) for Bayes error, how would you define “human-level performance”?

To find a reasonable proxy for Bayes error, you should clearly identify what the purpose of your human level performance should be.
For example, since doctors have an error of 0.5%, in a given problem we would define 0.5% as the human level error as a proxy for Bayes error. 
The more specific in what you are defining, the better your outcome, e.g., the error rate by doctors who are above 50 years old and specialize in brain surgery.

Human Level Proxy for Bayes Error

Which of the following statements about algorithmic performance do you agree with?

Bayes Classifier is the probabilistic model that makes the most probable prediction for new examples given the training data. Bayes error rate is the minimum possible error that can be made when making predictions.

Bayes Calssifier and Bayes Error Rate

In statistical classification, Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error.

For deep learning, human-level performance can be regarded approximately as Bayes error rate.

University of Michigan - Ann Arbor

- Confusion Matrix
- False-Positive rate (Type-I error)
- False-Negative rate (Type-II error)
- Accuracy
- Recall
- F1 Score
- Optimizing vs. Satisficing
- Specificity (True Negative rate Specificity)
- Negative Predictive Value
- False Discovery Rate
- True-Positive rate (Recall, Sensitivity)
- Positive-Predictive Value (Precision)
- ROC curve and ROC AUC

Common Performance Metrics for Classification

The confusion matrix is a performance measurement method that can be used for binary classification and multi-class classifications.  It is usually shown shown in a table format with the four possible outcomes: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).  

Confusion Matrix

ROC stands for "receiver operating characteristics," a name derived from communications theory, the field in which ROC curves were first used. 

The ROC curve illustrates how false positive and true positive rates for a classifier vary as you change the threshold of the classifier. In the sample graph the threshold values are not shown, but a threshold value corresponding to a point in the upper-left of the graph would likely optimize your classifier. Such a threshold value would make your classifier predict many positive observations as positive (high true positive rate), and not predict many negative observations as positive (low false positive rate). The “steepness” of the curve maximizes true positive rate while minimizing the false positive rate 

AUC simply stands for "area under curve". 

Generally, higher the ROC AUC for a classifier, the better it is. This is because having more threshold values for which the true positive rate is higher than the false positive rate and having larger ratios of true positive rate to false positive rate (both good things) increases the ROC AUC. ROC AUC can take on values between 0 and 1. A value close to 1 means the classifier is quite good. A value of 0.5 means the classifier is no better than guessing randomly. The blue line on the sample graph has an AUC of 0.95 and the dotted line has an AUC of 0.5. 

ROC Curve and ROC AUC

https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall 

Precision and Recall performance metrics.

In statistical analysis of a binary classification, the F1 score is a measure of a test's accuracy, defined as the average of precision and recall. It is the harmonic mean of the precision and recall, with the highest possible value being 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either the precision or the recall is zero.

Some applications for using F1 score include measuring search, document classification, and query classification performance. The F-score has been widely used in the natural language processing literature, such as in the evaluation of named entity recognition and word segmentation.

Some common criticisms of F1 score include that it gives equal importance to recall and precision. It also does not take into account true negatives, thus making it susceptible to unbalanced class bias.

$$F_1 = 2\frac{1}{(\frac{1}{precision}+\frac{1}{recall})}$$

F1 Score

The criteria by which you expect your classification model to do as best as possible.

Optimizing Criteria in Classification Problems

The criteria by which you expect your classification model to do better than some threshold.

Satisficing Criteria in Classification Problems

Bayes error rate

In a future society, a machine is used to predict a crime before it occurs. If you were responsible to tuning this machine, how would you ensure no innocent people (people not about to commit a crime) are imprisoned (where crime is the positive label)?

What evaluation metric would you want to maximize based on the following scenario?

Recall, also known as the True Positive Rate (TPR), is defined as True Positives / (True Positives + False Negatives), i.e., what % of actual outcomes (labels) are correctly recognized.

Recall is important in tasks that required as many possible positives to be identified.

Recall is especially important in medicinal and legal applications (ex: fraud detection and tumor detection), and it is occasionally paired with someone who manually filters out all of the False Positives. Recall is also related to F1 Score and ROC Curves.


Recall of a Classification Model

Precision, also known as Positive-Predictive Value, is an important evaluation metric and is defined as True Positives / (True Positives + False Positives), i.e., among recognized outcomes (labels), what % are correct (actual) outcomes.

This is important especially when tasks need to be certain that a positive prediction is actually a True Positive. 

Precision is important in many tasks that directly impact consumers (as a False Positive would not be good). Some examples include search engine classification, sentiment analysis, and movie/item recommendation services.



Precision of a Classification Model

How much does accuracy (or other metric) change as a function of key learning parameter values?

Sensitivity Analysis of a Classification Model

How much does accuracy (or other metric) change as a function of the amount of training data?

Learning Curve of a Classification Model

Having three evaluation metrics makes it harder for you to quickly choose between two different algorithms, and will slow down the speed with which your team can iterate. True/False?

- We need an algorithm that can be as accurate as possible.
- We want the trained model to take no more than 10sec to classify a new image.
- We want the model to fit in 10 MB of memory.

If you had the four following models, which one would you choose based on the following accuracy, runtime, and memory size criteria?

Coverage is a fraction of the data set for which the machine learning system makes a prediction and produces a response. 

Coverage

According to the formula, since we always want to maximize True Positive, the key point is whether we want to minimize False Positive (choose precision) or minimize False Negative (choose recall). 

Cases where we want to minimize FP include crime detection, recommendation and so on, where we don't want to falsely detect a positive situation.

Cases where we want to minimize FN include prisoning, tumor detection and so on. The cost of falsely predicting a negative situation is very large.

However, the final decision depends on the actual needs that are specified.

How to choose between precision and recall?

F-Measure is a metric that incorporates aspects of both precision and recall, defined as:
$$F_\beta = \frac{(\beta^2 + 1)PR}{\beta^2P + R}$$
The $\beta$ parameter differentially weights the importance of recall $R$ and precision $P$, based perhaps on the needs of an application. Values of $\beta > 1$ favor recall, while values of $\beta < 1$ favor precision. When $\beta = 1$, the measure is simply the F-1 score, where precision and recall are equally balanced.

Learn Before

Related

Learn After