The most common way of conducting a human evaluation is to use a questionnaire that is rated on a numerical scale. The second most typical evaluation method is based on preference. Here the outputs are preferred or ranked in relation to each other. There are also qualitative evaluation, presenting binary questions or conducting user-study, 

University of Michigan - Ann Arbor

Human evaluation in natural language generation (NLG) has become a hot topic lately, with the emergence of several survey papers on the topic that study how human evaluation has been conducted in the past in the field of NLG in general.

Not all natural language generation tasks are of the nature that they are designed to convey factual information. Some of the NLG tasks deal with producing text of aesthetic nature such as poetry, stories, humor and so on. We call these creative NLG tasks. 

These types of tasks are simultaneously researched in two distinct fields of science: natural language processing (NLP) and computational creativity (CC).

Human evaluation in natural language generation (NLG)

Hämäläinen, M., & Alnajjar, K. (2021). Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers. arXiv preprint arXiv:2108.00308.

Human Evaluation of Creative NLG Systems: An Interdisciplinary Survey on Recent Papers

The evaluated parameters are very different for the large range of NLG tasks. The parameters include meaning, syntactic correctness of the language, novelty, relevance and emotional value.

NLG Human Evaluation features

Ways to conduct evaluation

Even work on closely related topics prefers to use their own evaluation methods that are not based on any existing research. If the evaluation is based on existing research, the evaluation questions are not motivated in the earlier research either. This type of evaluation has become to be known as a symptom of the Great Misalignment Problem.

Apart from the evaluation questions not aligning with the model, a much larger problem related to evaluation questions can be identified. Most of the papers were not clear about the actual evaluation questions used, instead they listed the evaluated parameters as though human evaluation was like an automated one where one can just score abstract notions such as typicality or fluency accurately on a 5 point scale.

Problems with evaluation questions

The ways of how human evaluation of creative NLG systems should be conducted:

- Define the goals: Once the goals are clearly stated, it is easy to see the degree to which your implementation solution tries to achieve those goals, and how much can be attributed to the method and how much to the training data.

- Go concrete: By using evaluation questions that are as concrete as possible, you can reduce the room for subjective interpretation.

- Run some tests: The same concept can be evaluated through multiple different wordings. It is better to adjust your evaluation questions sooner than after running a costly crowd-sourcing.

- Run multiple evaluations

- Report everything clearly

- Analyze your results

Learn Before

Related