Question: How can a practitioner use human annotations to approximate the lowest error rate a model could realistically achieve? Describe when this approach is appropriate and what comparison should be made.

Sample answer: This method works best for tasks that people can do reliably without special equipment or training, such as classifying common objects in photos, spotting obvious defects in product images, or converting clear spoken words into text. In such cases, the practitioner should ask a person to label a representative set of examples and then measure how often those labels match the dataset’s ground-truth labels. That human-vs.-ground-truth accuracy provides an estimate of the task’s practical ceiling, which in turn implies a lower bound on the achievable error rate.

Key points:

• Use this on tasks people can perform well
• Have a human label a sample of the data
• Compare the human labels with the ground-truth labels to measure accuracy
• Convert that accuracy into an estimate of the minimum likely error rate

Rubric: A full-credit answer must state that the approach is appropriate for tasks humans can do well, explain that a human should label sample examples, and specify that those labels must be compared against ground-truth labels to estimate accuracy and thus the best achievable error rate.