Why Comparing to Human-Level Performance Helps ML Development
Building an ML system is easier for tasks people do well because human labelers can provide data, error analysis can draw on human intuition, and human-level performance can estimate the optimal error rate and set a desired error rate. A reasonable, achievable target error rate can accelerate team progress, and knowing the algorithm has high avoidable bias opens up a menu of improvement options.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Why Comparing to Human-Level Performance Helps ML Development
When a system has 40% error, which human-level reference is most useful for guiding improvement?
When a system's error rate is 40%, choosing a junior doctor (10% error) vs. an experienced doctor (5% error) as the human-level reference makes little practical difference.
If your system is already at _____ % error, defining the human-level reference as 2% gives better tools to keep improving.
Match each system error scenario to its implication for selecting a human-level reference.
Order the reasoning steps for deciding whether precision in the human-level reference matters for your system.
Why does a 2% human-level reference give better improvement tools when a system is at 10% error, compared to when it is at 40% error?
At high system error rates such as 40%, a more precise human-level reference always provides significantly better guidance than a less precise one.
According to Machine Learning Yearning, human error rate can be used as the _____ error rate when hoping for human-level performance.
Match each annotator type or system scenario to its description from Machine Learning Yearning.
Order the steps in the decision process for selecting an appropriate human-level error reference for an ML system.