In adversarial evaluation, a classifier is trained to distinguish between human and machine responses; the better a chatbot is at fooling the classifier, the more effective it is.

Adversarial Evaluation

An example of an observer evaluation where an observer looks at two separate human-computer conversations side-by-side, and chooses which one performed better based on engagingness, interestingness, humanness, and knowledgeability. 

Acute-Eval Metric

Chatbots are typically evaluated by humans, who assign the chatbot a score that weighs different characteristics of performance such as engagingness,  fluency, humanness, etc. Chatbots can be evaluated by either a user (participant evaluation) or a third-party observer (observer evaluation).

Carleton College

Chatbots are designed for unstructured conversation, and are meant to mimic natural back and forth human dialogue

Chatbots

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

Rule-based chatbots are chatbots that create their conversations from set rules, such as responding "What makes you think I _____ you" in response to the statement "You _____ me." Early notable rule-based chatbots include ELIZA, which essentially acted as a therapist to clinical psychology patients, and PARRY, which modeled schizophrenic behavior using responses based on variables for fear and anger.

Rule-Based Chatbots

Corpus-based chatbots are chatbots that draw their conversational data from corpuses, or vast linguistic datasets, of human conversation from various sources like social media and TV shows. 

Corpus-Based Chatbots

Evaluating Chatbots

Chatbot's architecture can be hybrid ,taking advantage of both rule-based and neural/corpus based systems sometimes they even utilize elements from frame-based structure.

Hybrid Chatbots

ELIZA is an early natural language processing system that could carry on a limited conversation with a user by imitating the responses of a Rogerian psychotherapist.

Learn Before

Related

Learn After