In order to judge the efficacy of task-based dialogue systems, users can fill out evaluations after performing a task. However, we can't run these after every system change, as it would be too much work.

University of California, Santa Cruz

- Absolute task success (for simple, ambiguous tasks; was it done correctly?)
- User satisfaction rating
- Task completion success
- Task error rate
- Efficiency cost

Ways to Evaluate Task-Based Dialogue Agents

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

User Satisfaction Ratings

Task completion success is a measure of the total correctness of a solution. This can include the  **slot error rate** (also known as the concept error rate), which is the percentage of slots filled with correct values per sentence.

*slot error rate* = $\frac{inserted,  deleted,  or  substituted slots}{total  reference  slots}$ 

Often, however, a user's perception of whether they completed the task is a better predictor of satisfaction than actual task completion success.

Task Completion Success

A task's task error rate measures how often a test is done correctly. This can include:
- Efficiency cost: measures of a system's efficiency at helping users, like total elapsed time to complete a task, total queries, total turns/system turns, system non-responses, or turn-correction ratio (the number of system/user turns used solely to correct errors over the number of turns)

- Quality cost: measures other aspects that affect the user's perception of system quality, such as number of manual interrupts and failures to return text output. It can also focus on how well the system understood and responded to the user, such as the slot error rate mentioned before.

Learn Before

Related