Why Comparing to Human-Level Performance Helps ML Development

When a system has 40% error, which human-level reference is most useful for guiding improvement?

When a system's error rate is 40%, choosing a junior doctor (10% error) vs. an experienced doctor (5% error) as the human-level reference makes little practical difference.

If your system is already at _____ % error, defining the human-level reference as 2% gives better tools to keep improving.

Match each system error scenario to its implication for selecting a human-level reference.

Order the reasoning steps for deciding whether precision in the human-level reference matters for your system.

Why does a 2% human-level reference give better improvement tools when a system is at 10% error, compared to when it is at 40% error?

At high system error rates such as 40%, a more precise human-level reference always provides significantly better guidance than a less precise one.

According to Machine Learning Yearning, human error rate can be used as the _____ error rate when hoping for human-level performance.

Match each annotator type or system scenario to its description from Machine Learning Yearning.

Order the steps in the decision process for selecting an appropriate human-level error reference for an ML system.