MRR is often used to evaluate the performance of ranking algorithms in NLP
tasks such as query-document ranking and QA. MRR is defined below, where 𝑄 is a set of all possible answers, and 𝑟𝑎𝑛𝑘(𝑖) is the ranking position of the ground-truth answer.


Saint Olaf College

5 main methods: Accuracy and Error Rate; Precision / Recall / F1 score; Exact Match (EM); Mean Reciprocal Rank (MRR)

Text classification performance analysis

These are primary metrics to evaluate the quality of a classification model. Let TP,
FP, TN, and FN denote true positive, false positive, true negative, and false negative. The classification Accuracy and Error Rate are defined in the image below.

Accuracy and Error Rate

These are also primary metrics and are more often used than accuracy or error
rate for imbalanced test sets, e.g., the majority of the test samples have one class label. Precision and recall for binary classification are defined as below. The F1 score is the harmonic mean of the precision and recall, as in Eq. 3. An F1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.

Precision / Recall / F1 score

The exact match metric is a popular metric for question-answering systems, which
measures the percentage of predictions that match any one of the ground truth answers exactly. EM is one of the main metrics used for SQuAD.

Learn Before

Related