Question: Explain how to establish a human performance benchmark for a specific subset of data, such as delivery-room X-ray images.

Sample answer: To measure human performance for one exact data set, you must use examples drawn from that same data distribution rather than from a different one. For example, if the benchmark is for delivery-room X-ray images, you should collect X-rays from that setting and have people label those images directly. After that, calculate the labelers’ error rate on that specific set. That error rate is the human benchmark for the target distribution.

Key points:

• Use examples from the exact target data set.
• Ask humans to do the same task on that data.
• Compute the error rate on that target set.

Rubric: A complete response should state that the benchmark must be measured on the target distribution itself, that humans should perform the labeling task on those examples, and that the resulting error rate is the benchmark.