University of Michigan - Ann Arbor

Google

- Macro-average: each class has equal weight. 
1. Compute metric with each class 
2. Average resulting metrics across classes 
- Micro-average: Each instance has equal weight, largest class has most influence
1. Aggregate outcomes across classes
2. Compute metric with aggregate outcomes 

If want to weight metric towards largest classes → use micro-averaging
If you want to weight metric towards smallest classes → use macro-averaging 


Multi-Class Evaluation Metrics

1. Confusion Matrix
2. Accuracy, Recall, Precision
3. F1 Score
4. Receiving Operating Characteristics(ROC),Area under the Curve(AUC)
5. Log Loss

Evaluation Metrics of Classification Models

- Confusion Matrix
- False-Positive rate (Type-I error)
- False-Negative rate (Type-II error)
- Accuracy
- Recall
- F1 Score
- Optimizing vs. Satisficing
- Specificity (True Negative rate Specificity)
- Negative Predictive Value
- False Discovery Rate
- True-Positive rate (Recall, Sensitivity)
- Positive-Predictive Value (Precision)
- ROC curve and ROC AUC

Common Performance Metrics for Classification

Typically, the $$R^2$$ score is sufficient for evaluating a regression model. Alternative evaluation metrics include:
- Mean Absolute Error (MAE): The absolute difference between target and predicted values, representing the expected value of $$L_1$$ norm loss.
- Mean Squared Error (MSE): The squared difference between target and predicted values, representing the expected value of $$L_2$$ norm loss.
- Median Absolute Error: Uses the median of the error distribution rather than the mean, making it more robust to outliers.

Characteristics of Regression Evaluation Metrics

Regression model evaluation metrics are quantitative measures used to assess how well a model's predictions match the actual observed data. Common metrics include Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), the Coefficient of Determination ($$R^2$$), and Adjusted $$R^2$$.

Regression Model Evaluation Metrics

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

https://datascience.stackexchange.com/questions/15989/micro-average-vs-macro-average-performance-in-a-multiclass-classification-settin

Micro Average vs Macro average Performance in a Multiclass classification setting

http://notesbyanerd.com/2014/12/17/multi-class-performance-measures/

Multi-class Performance Measures

Macro-average is the average of some metric (e.g. precision or recall) among classes, where all classes are weighted equally.  Since this effectively gives a "boost" to smaller classes, macro-averages are more indicative of the precision/recall within smaller classes than within larger classes. According to the referenced article, however, macro-averaging can be bad practice when classes vary widely in their number of instances.

Macro-average

Micro-average is the average of some metric (e.g. precision or recall) among classes, where classes are weighted proportionally by their number of instances. Another way of saying this is that all instances are weighted equally. Since larger classes have more instances and all instances have equal weight, micro-averages are more indicative of the precision/recall within larger classes. In other words, larger classes have more influence.

Micro-average

Having three evaluation metrics makes it harder for you to quickly choose between two different algorithms, and will slow down the speed with which your team can iterate. True/False?

In one-to-many model training, one category is used as a positive example, and the remaining examples are used as negative examples to train K models; when making predictions, if one of the K models is output as a positive example, then the final The prediction result belongs to this category of the classifier; if multiple positive examples are generated, the confidence of the classifier can be selected as the indicator, and the classifier with the highest confidence is selected as the final prediction result. Common confidences are accuracy score and recall rate.

OVA/OVR (One-Versus-All/One-Versus-the-Rest)

Divide the model construction application into two stages: encoding and decoding stages;
Coding stage: perform M division among K categories. Each division divides part of the data into positive categories and part of the data into negative categories. Each division builds a model. The result of the model is for each Each category defines a point;
Decoding stage: Use the trained model to predict the test sample, define the sample to be predicted as a point in the space, calculate the distance between the point to be tested and the category, and select the category with the closest distance as the final forecast category.

Error Correcting Output Codes

One-Versus-One (OVO) is a strategy for extending binary classifiers to solve multi-class classification problems. For a dataset with $$K$$ classes, it trains $$K(K-1)/2$$ separate binary classifiers, one for each possible pair of classes. During prediction, a test instance is evaluated by all classifiers, and the class that receives the most predictions (majority voting) is chosen as the final output.

One-Versus-One (OVO) Classification

After training a classificaiton model, we can measure the accuracy of a model by predicting the outcome of the observations in the test set.  If the model cannot predictive expected outcome, the accuracy is lower, and the opposite is true for the correct predictions.  

Clasification Accuracy

Formulas for evaluation


Predict_proba
-typical rule: choose most like class
-Adjusting threshold affeccts predictions of classifier
-Higher threshold results in a more conservative classifier (e.g. only predict class I if estimated probability of class I is above 70%)
-Not all models provide realistic probability estimates

Predicted Probability of class membership(predict_proba)

The confusion matrix is a performance measurement method that can be used for binary classification and multi-class classifications.  It is usually shown shown in a table format with the four possible outcomes: True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN).  

Confusion Matrix

ROC stands for "receiver operating characteristics," a name derived from communications theory, the field in which ROC curves were first used. 

The ROC curve illustrates how false positive and true positive rates for a classifier vary as you change the threshold of the classifier. In the sample graph the threshold values are not shown, but a threshold value corresponding to a point in the upper-left of the graph would likely optimize your classifier. Such a threshold value would make your classifier predict many positive observations as positive (high true positive rate), and not predict many negative observations as positive (low false positive rate). The “steepness” of the curve maximizes true positive rate while minimizing the false positive rate 

AUC simply stands for "area under curve". 

Generally, higher the ROC AUC for a classifier, the better it is. This is because having more threshold values for which the true positive rate is higher than the false positive rate and having larger ratios of true positive rate to false positive rate (both good things) increases the ROC AUC. ROC AUC can take on values between 0 and 1. A value close to 1 means the classifier is quite good. A value of 0.5 means the classifier is no better than guessing randomly. The blue line on the sample graph has an AUC of 0.95 and the dotted line has an AUC of 0.5. 

ROC Curve and ROC AUC

https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall 

Precision and Recall performance metrics.

In statistical analysis of a binary classification, the F1 score is a measure of a test's accuracy, defined as the average of precision and recall. It is the harmonic mean of the precision and recall, with the highest possible value being 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either the precision or the recall is zero. Some applications for using F1 score include measuring search, document classification, and query classification performance. The F-score has been widely used in the natural language processing literature, such as in the evaluation of named entity recognition and word segmentation. Some common criticisms of F1 score include that it gives equal importance to recall and precision. It also does not take into account true negatives, thus making it susceptible to unbalanced class bias. $$F_1 = 2\frac{1}{(\frac{1}{precision}+\frac{1}{recall})}$$

F1 Score

The criteria by which you expect your classification model to do as best as possible.

Optimizing Criteria in Classification Problems

The criteria by which you expect your classification model to do better than some threshold.

Satisficing Criteria in Classification Problems

In a future society, a machine is used to predict a crime before it occurs. If you were responsible to tuning this machine, how would you ensure no innocent people (people not about to commit a crime) are imprisoned (where crime is the positive label)?

What evaluation metric would you want to maximize based on the following scenario?

Recall, also known as the True Positive Rate (TPR), is defined as True Positives / (True Positives + False Negatives), i.e., what % of actual outcomes (labels) are correctly recognized.

Recall is important in tasks that required as many possible positives to be identified.

Recall is especially important in medicinal and legal applications (ex: fraud detection and tumor detection), and it is occasionally paired with someone who manually filters out all of the False Positives. Recall is also related to F1 Score and ROC Curves.


Recall of a Classification Model

Precision, also known as Positive-Predictive Value, is an important evaluation metric and is defined as True Positives / (True Positives + False Positives), i.e., among recognized outcomes (labels), what % are correct (actual) outcomes.

This is important especially when tasks need to be certain that a positive prediction is actually a True Positive. 

Precision is important in many tasks that directly impact consumers (as a False Positive would not be good). Some examples include search engine classification, sentiment analysis, and movie/item recommendation services.



Precision of a Classification Model

How much does accuracy (or other metric) change as a function of key learning parameter values?

Sensitivity Analysis of a Classification Model

How much does accuracy (or other metric) change as a function of the amount of training data?

Learning Curve of a Classification Model

- We need an algorithm that can be as accurate as possible.
- We want the trained model to take no more than 10sec to classify a new image.
- We want the model to fit in 10 MB of memory.

If you had the four following models, which one would you choose based on the following accuracy, runtime, and memory size criteria?

Coverage is a fraction of the data set for which the machine learning system makes a prediction and produces a response. 

Coverage

According to the formula, since we always want to maximize True Positive, the key point is whether we want to minimize False Positive (choose precision) or minimize False Negative (choose recall). 

Cases where we want to minimize FP include crime detection, recommendation and so on, where we don't want to falsely detect a positive situation.

Cases where we want to minimize FN include prisoning, tumor detection and so on. The cost of falsely predicting a negative situation is very large.

However, the final decision depends on the actual needs that are specified.

How to choose between precision and recall?

F-Measure is a metric that incorporates aspects of both precision and recall, defined as:
$$F_\beta = \frac{(\beta^2 + 1)PR}{\beta^2P + R}$$
The $\beta$ parameter differentially weights the importance of recall $R$ and precision $P$, based perhaps on the needs of an application. Values of $\beta > 1$ favor recall, while values of $\beta < 1$ favor precision. When $\beta = 1$, the measure is simply the F-1 score, where precision and recall are equally balanced.

F-Measure

In statistical classification, the Bayes error rate is the lowest possible error rate for any classifier of a random outcome (into, for example, one of two categories) and is analogous to the irreducible error. In deep learning, human-level performance is often used as a proxy for the Bayes error rate.

Bayes Error Rate

When dealing with financial datasets such as stock or house prices, practitioners generally care more about the relative error $$\frac{y - \hat{y}}{y}$$ than the absolute error $$y - \hat{y}$$. For instance, a prediction error of \$100,000 is massive when estimating the price of a typical \$125,000 rural house, but the exact same absolute error represents a stunningly accurate prediction for a median house price exceeding \$4 million in a wealthy neighborhood.

Learn Before

Related