In order to judge the efficacy of task-based dialogue systems, users can fill out evaluations after performing a task. However, we can't run these after every system change, as it would be too much work.

University of California, Santa Cruz

Google

- Absolute task success (for simple, ambiguous tasks; was it done correctly?)
- User satisfaction rating
- Task completion success
- Task error rate
- Efficiency cost

Ways to Evaluate Task-Based Dialogue Agents

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

User Satisfaction Ratings

Task completion success is a measure of the total correctness of a solution provided by a task-oriented dialogue agent. Often, a user's perception of whether they completed the task is a better predictor of user satisfaction than actual objective task completion success.

Task Completion Success

A task's task error rate measures how often a test is done correctly. This can include:
- Efficiency cost: measures of a system's efficiency at helping users, like total elapsed time to complete a task, total queries, total turns/system turns, system non-responses, or turn-correction ratio (the number of system/user turns used solely to correct errors over the number of turns)

- Quality cost: measures other aspects that affect the user's perception of system quality, such as number of manual interrupts and failures to return text output. It can also focus on how well the system understood and responded to the user, such as the slot error rate mentioned before.

Learn Before

Related