Question: Discuss how the current error rate of a machine learning system impacts the selection of a human-level reference for labeling data and providing intuition. Explain the difference in approach when a system has a very high error rate versus when it is approaching typical human performance, using the example of junior versus experienced doctors to illustrate your point.

Sample answer: When a system has a high error rate, such as 40%, the gap to human-level performance is large enough that using a junior doctor (10% error) or an experienced doctor (5% error) as a reference makes little practical difference for guiding improvements. However, as the system improves and reaches lower error rates like 10%, a more precise human-level reference, such as a 2% error rate from a highly experienced source, becomes necessary. This tighter reference provides better tools and clearer signals to continue improving the system when the margins for error reduction are smaller.

Key points:

• High system error (e.g., 40%) means the precision of the human reference (10% vs 5%) matters very little.
• Low system error (e.g., 10%) requires a much more precise human reference (e.g., 2%).
• A more precise human reference provides better tools to keep improving a highly performing system.

Rubric: Full credit is given for explaining that high system error requires less precise human references, while lower system error demands highly precise references to provide adequate tools for continued improvement, supported by the medical expert example.