In order to judge the efficacy of task-based dialogue systems, users can fill out evaluations after performing a task. However, we can't run these after every system change, as it would be too much work.

User Satisfaction Ratings

Task completion success is a measure of the total correctness of a solution provided by a task-oriented dialogue agent. Often, a user's perception of whether they completed the task is a better predictor of user satisfaction than actual objective task completion success.

Task Completion Success

A task's task error rate measures how often a test is done correctly. This can include:
- Efficiency cost: measures of a system's efficiency at helping users, like total elapsed time to complete a task, total queries, total turns/system turns, system non-responses, or turn-correction ratio (the number of system/user turns used solely to correct errors over the number of turns)

- Quality cost: measures other aspects that affect the user's perception of system quality, such as number of manual interrupts and failures to return text output. It can also focus on how well the system understood and responded to the user, such as the slot error rate mentioned before.

Task Error Rate

- Absolute task success (for simple, ambiguous tasks; was it done correctly?)
- User satisfaction rating
- Task completion success
- Task error rate
- Efficiency cost

University of California, Santa Cruz

Google

Task-oriented dialogue agents are designed to aid users in completing structured tasks, such as answering questions and making bookings

Task-Oriented Dialogue Agents

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

It's a task-oriented dialogue system. It is intended to engage a sympathetic and highly cooperative human in an English dialog, directed towards a specific goal within a very restricted domain of discourse.

Genial Understander System(GUS)

It's also called domain ontology. It's a kind of knowledge structure representing the kinds of intentions that the system can extract from user sentences and consist of a collection of slots, each of which can take a set of possible values.

Learn Before

Related

Learn After