MRR is designed for systems that return a short ranked list of answers or passages for each test set question, which we can compare against the (human-labeled) correct answer. MRR is computed by the formula below:

Q = test set of questions

rank = rank of the first correct answer

MRR = sum of 1 / rank for all the questions in Q / size of Q

Hamilton College

Common approach(es) to evaluating Factoid question answering and reading comprehension systems.

Evaluation of Answers

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Mean Reciprocal Rank (MRR)

Exact match is one of the two metrics used to evaluate reading comprehension systems. 

Exact Match = The % of predicted answers that match the gold answer exactly.

Exact Match

F1 score is one of the metrics used to evaluate reading comprehension systems.
It takes the average word/token overlap between predicted and gold answers. Treat the prediction and gold as a bag of tokens, and compute F1 for each question, then return the average F1 over all questions.

Learn Before

Related