An example of an observer evaluation where an observer looks at two separate human-computer conversations side-by-side, and chooses which one performed better based on engagingness, interestingness, humanness, and knowledgeability. 

Carleton College

Chatbots are typically evaluated by humans, who assign the chatbot a score that weighs different characteristics of performance such as engagingness,  fluency, humanness, etc. Chatbots can be evaluated by either a user (participant evaluation) or a third-party observer (observer evaluation).

Evaluating Chatbots

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

In adversarial evaluation, a classifier is trained to distinguish between human and machine responses; the better a chatbot is at fooling the classifier, the more effective it is.

Learn Before

Related