Using a Human Benchmark to Choose Where to Improve a Model
Question: Why is it useful to compare each stage of a machine learning system with expert human performance before choosing what to improve next? Explain why a stage that is far below the human benchmark often deserves priority.
Sample answer: A human benchmark gives a practical reference point for judging how much room a system still has to improve. When one stage is much worse than expert human performance, that gap suggests there is still substantial headroom and that better results may be achievable with focused work. By contrast, if a stage is already close to the human benchmark, further gains are usually smaller and harder to obtain. For that reason, engineers often prioritize the component with the largest gap because effort spent there is more likely to produce meaningful improvement.
Key points:
- Human performance provides a reference for estimating remaining improvement potential.
- A large gap usually signals a promising target with more attainable gains.
- Prioritizing the weakest stage helps avoid spending effort where returns are likely to be small.
Rubric: The response should explain 1) that human performance serves as a benchmark, 2) that a large gap indicates higher improvement potential and thus higher priority, and 3) that this approach helps avoid overinvesting in stages that are already near the benchmark.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What is the best response when one stage in a multi-step system is far worse than expert performance?
True or False: If a subsystem performs much worse than expert-level performance on the same task, it is a strong candidate for targeted improvement.
If a pipeline stage is far from _____ performance, it is a strong candidate for improvement.
Match each model-status description to the right improvement priority.
Order the steps for using human-level comparisons to decide which subsystem to improve first.
Why should a subsystem far below expert-level performance usually receive top attention?
True or False: The best place to focus improvement is a module that already matches human performance closely.
When one module is far worse than human performance, you have a strong _____ to improve that module.
Match each model-performance scenario to the most appropriate takeaway.
Order the reasoning chain for choosing the weakest stage in a multi-step system to improve first.
Using a Human Benchmark to Choose Where to Improve a Model
Choosing the Bottleneck in a Fraud-Review Pipeline
When to prioritize improving one pipeline stage