Question: Explain the methodology for establishing a human-level performance benchmark for a specific data subset, such as mobile cat images.

Sample answer: To establish a human-level performance benchmark for a specific data subset like mobile images, you cannot rely on human performance on a different distribution (like internet images). Instead, you must gather examples specifically from the target distribution (mobile cat images). You then ask humans to perform the task—in this case, labeling the images. Finally, you measure the error rate of the human labelers on this specific dataset. This resulting error rate represents the human-level performance for that exact data distribution.

Key points:

• Use data from the specific target distribution (e.g., mobile images).
• Ask humans to label this specific data.
• Measure the resulting human error rate on the target data.

Rubric: A strong answer will explicitly mention using the specific target data distribution, having humans perform the labeling task on that data, and calculating their resulting error rate.