The confusion matrix is a performance measurement method that can be used for binary classification and multi-class classifications.  It is usually shown shown in a table format with the four possible outcomes: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).  

Confusion Matrix

ROC stands for "receiver operating characteristics," a name derived from communications theory, the field in which ROC curves were first used. 

The ROC curve illustrates how false positive and true positive rates for a classifier vary as you change the threshold of the classifier. In the sample graph the threshold values are not shown, but a threshold value corresponding to a point in the upper-left of the graph would likely optimize your classifier. Such a threshold value would make your classifier predict many positive observations as positive (high true positive rate), and not predict many negative observations as positive (low false positive rate). The “steepness” of the curve maximizes true positive rate while minimizing the false positive rate 

AUC simply stands for "area under curve". 

Generally, higher the ROC AUC for a classifier, the better it is. This is because having more threshold values for which the true positive rate is higher than the false positive rate and having larger ratios of true positive rate to false positive rate (both good things) increases the ROC AUC. ROC AUC can take on values between 0 and 1. A value close to 1 means the classifier is quite good. A value of 0.5 means the classifier is no better than guessing randomly. The blue line on the sample graph has an AUC of 0.95 and the dotted line has an AUC of 0.5. 

ROC Curve and ROC AUC

https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall 

Precision and Recall performance metrics.

In statistical analysis of a binary classification, the F1 score is a measure of a test's accuracy, defined as the average of precision and recall. It is the harmonic mean of the precision and recall, with the highest possible value being 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either the precision or the recall is zero.

Some applications for using F1 score include measuring search, document classification, and query classification performance. The F-score has been widely used in the natural language processing literature, such as in the evaluation of named entity recognition and word segmentation.

Some common criticisms of F1 score include that it gives equal importance to recall and precision. It also does not take into account true negatives, thus making it susceptible to unbalanced class bias.

$$F_1 = 2\frac{1}{(\frac{1}{precision}+\frac{1}{recall})}$$

F1 Score

The criteria by which you expect your classification model to do as best as possible.

Optimizing Criteria in Classification Problems

The criteria by which you expect your classification model to do better than some threshold.

Satisficing Criteria in Classification Problems

In statistical classification, Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error.

For deep learning, human-level performance can be regarded approximately as Bayes error rate.

Bayes error rate

In a future society, a machine is used to predict a crime before it occurs. If you were responsible to tuning this machine, how would you ensure no innocent people (people not about to commit a crime) are imprisoned (where crime is the positive label)?

What evaluation metric would you want to maximize based on the following scenario?

Recall, also known as the True Positive Rate (TPR), is defined as True Positives / (True Positives + False Negatives), i.e., what % of actual outcomes (labels) are correctly recognized.

Recall is important in tasks that required as many possible positives to be identified.

Recall is especially important in medicinal and legal applications (ex: fraud detection and tumor detection), and it is occasionally paired with someone who manually filters out all of the False Positives. Recall is also related to F1 Score and ROC Curves.


Recall of a Classification Model

Precision, also known as Positive-Predictive Value, is an important evaluation metric and is defined as True Positives / (True Positives + False Positives), i.e., among recognized outcomes (labels), what % are correct (actual) outcomes.

This is important especially when tasks need to be certain that a positive prediction is actually a True Positive. 

Precision is important in many tasks that directly impact consumers (as a False Positive would not be good). Some examples include search engine classification, sentiment analysis, and movie/item recommendation services.



Precision of a Classification Model

How much does accuracy (or other metric) change as a function of key learning parameter values?

Sensitivity Analysis of a Classification Model

How much does accuracy (or other metric) change as a function of the amount of training data?

Learning Curve of a Classification Model

Having three evaluation metrics makes it harder for you to quickly choose between two different algorithms, and will slow down the speed with which your team can iterate. True/False?

- We need an algorithm that can be as accurate as possible.
- We want the trained model to take no more than 10sec to classify a new image.
- We want the model to fit in 10 MB of memory.

If you had the four following models, which one would you choose based on the following accuracy, runtime, and memory size criteria?

Coverage is a fraction of the data set for which the machine learning system makes a prediction and produces a response. 

Coverage

According to the formula, since we always want to maximize True Positive, the key point is whether we want to minimize False Positive (choose precision) or minimize False Negative (choose recall). 

Cases where we want to minimize FP include crime detection, recommendation and so on, where we don't want to falsely detect a positive situation.

Cases where we want to minimize FN include prisoning, tumor detection and so on. The cost of falsely predicting a negative situation is very large.

However, the final decision depends on the actual needs that are specified.

How to choose between precision and recall?

F-Measure is a metric that incorporates aspects of both precision and recall, defined as:
$$F_\beta = \frac{(\beta^2 + 1)PR}{\beta^2P + R}$$
The $\beta$ parameter differentially weights the importance of recall $R$ and precision $P$, based perhaps on the needs of an application. Values of $\beta > 1$ favor recall, while values of $\beta < 1$ favor precision. When $\beta = 1$, the measure is simply the F-1 score, where precision and recall are equally balanced.

F-Measure

- Confusion Matrix
- False-Positive rate (Type-I error)
- False-Negative rate (Type-II error)
- Accuracy
- Recall
- F1 Score
- Optimizing vs. Satisficing
- Specificity (True Negative rate Specificity)
- Negative Predictive Value
- False Discovery Rate
- True-Positive rate (Recall, Sensitivity)
- Positive-Predictive Value (Precision)
- ROC curve and ROC AUC

University of Michigan - Ann Arbor

Classification problems involve predicting a categorical label for a given input. The model's output for a classification task is typically a probability distribution over the set of possible labels, indicating the likelihood of the input belonging to each category.

Classification problem in statistical learning

James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning (Vol. 112, pp. 3-7). New York: springer.

An Introduction to Statistical Learning with Applications in R

Lantz, Brett (2015). Machine Learning with R - Second Edition: Expert techniques for predictive modeling to solve all your data analysis problems. Packt Publishing. 

Machine Learning with R (Second Edition)

There is a paper on predicting a user's personality from 16 personality types (based on the Myers-Briggs Type Index). The input values were the user's written text they provide on an Internet forum. This is a classification because the output is a qualitative prediction. For example, the user could be classified as an Extrovert.


Example of a classification problem in statistical learning

Classifiers are techniques used to predict qualitative variables.
- Methods that are not very computer intensive (widely-used)
- Methods that are much more computer intensive
- Decision Tree that can be used on regression and classification problems

Methods of Classification

1. Confusion Matrix
2. Accuracy, Recall, Precision
3. F1 Score
4. Receiving Operating Characteristics(ROC),Area under the Curve(AUC)
5. Log Loss

Evaluation Metrics of Classification Models

https://towardsdatascience.com/top-5-metrics-for-evaluating-classification-model-83ede24c7584

Classification Evaluation Metrics

What is the differnce between classification and regression?

In machine learning, classification refers to a predictive modeling problem where a class label is predicted for a given example of input data. There are three main types of classification tasks that you may encounter; they are:
- Binary Classification,
- Multi-Class Classification,
- Multi-Label Classification.


3 Types of Classification Tasks in Machine Learning

Imbalanced Classification refers to classification tasks where the number of examples in each class is unequally distributed. Typically, imbalanced classification tasks are binary classification tasks where the majority of examples in the training dataset belong to the normal class and a minority of examples belong to the abnormal class.

Balanced Classification is the opposite. It is primarily equally distributed and commonly occurs in multi-label or multi-class classification. 

Imbalanced Classification vs. Balanced Classification

Common Performance Metrics for Classification

Classification is essentially a function mapping input data to output categories. If there are missing data in the input, the function can perform poorly in the prediction task. Therefore, having multiple functions that do the predicting job collectively is crucial.

Classification with Missing Data

A data science team is building a predictive model for an e-commerce company. Which of the following tasks represents a classification problem?

Given the following scenario, explain why this task is a classification problem. In your explanation, describe the nature of the model's predicted output.

Learn Before

Related

Learn After