The character F-score (chrF) metric has several limitations. It is very local, meaning that a large phrase moved to a different position might barely change the chrF score. Furthermore, chrF cannot evaluate cross-sentence properties of a document, such as its discourse coherence. chrF and similar automatic overlap metrics also perform poorly when comparing fundamentally different systems, such as human-aided translation versus machine translation, or distinct machine translation architectures. Therefore, these metrics are most appropriate for evaluating changes within a single system.

University of Michigan - Ann Arbor

Google

The simplest and most robust metric for MT evaluation is called chrF, which stands for character F-score. Consider a test set from a parallel corpus, in which each source sentence has both a gold human target translation and a candidate MT translation we’d like to evaluate. The chrF metric ranks each MT target sentence by a function of the number of character n-gram overlaps with the human translation. (see pic)
Character or word overlap-based metrics like chrF are mainly used to compare two systems, with the goal of answering questions like: did the new algorithm we just invented improve our MT system?


chrF evaluation method

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Learn Before

Related