Scenario: A company is building a model that assigns common backyard bird photos to species. Experienced birdwatchers can usually identify these pictures without much difficulty, and the team wants a realistic estimate of how low the classification error could go.

Question: What procedure should the team use to estimate the best possible error rate for this photo classification task?

Sample answer: Ask knowledgeable birdwatchers to label a representative set of training examples. Then compare those human labels with the reference labels in the training data and use the observed human accuracy to approximate the error floor.

Key points:

• The problem is one that people can handle well with expertise
• Human experts should supply labels for the examples
• Human accuracy against the training labels gives a practical estimate of the lowest likely error

Rubric: The answer must recommend using expert human labeling and state that the optimal error rate is estimated by comparing those human labels with the training set labels.