Why Human Comparison Becomes Less Helpful After Strong Model Performance

When can human comparison still be useful after a model is already better than average human performance on the dev or test set?

Human judgment can still be useful on specific subsets even after a model beats average human performance overall.

When people still do better than the model on certain cases, they can provide better _____, useful intuition, and a target level of performance.

Match each benefit of comparing a model against people on a hard subset to its description.

Deciding What to Do After Beating Average Human Performance

In the document-reading example, on which task does the system beat people while people still do better on a different task?

If a classifier beats the average human score on the development set, then comparing its mistakes with human judgments is no longer useful for finding improvements.

Human-comparison methods remain useful as long as there are dev examples where people are _____ and the model is mistaken.

Match each item from the invoice-reading example to its role in the human-better-subset idea.

Order the steps for using a human-strong subset to improve a model.

How human comparison can still help a stronger-than-average system

Using a human-strong subset to improve a transaction review model

When Human-Versus-Model Comparisons Stop Helping