Why some error analysis methods need a human baseline
Question: Briefly explain why certain error analysis methods do not work well when a model contains steps that people cannot reliably do.
Sample answer: Many error analysis methods compare the system against what a person can do. If a component performs a task that people cannot do reliably, then there is no trustworthy human benchmark for that step. Without that reference point, those methods cannot tell whether the component is truly failing or simply operating in a task where human comparison is not meaningful.
Key points:
- Some error analysis methods depend on a human-performance benchmark.
- If people cannot do the task well, that benchmark does not exist.
- Without the benchmark, those methods are not useful for that component.
Rubric: The answer should mention the need for a human benchmark and explain that when humans cannot perform the task well, the benchmark is unavailable, so the related error analysis methods cannot be applied.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why is error analysis especially useful when the machine learning task is something people can already do reliably?
When people can perform a task reliably, their performance can be used as a reference point for analyzing model errors.
Human-Solvable Tasks and Better _____
Match each task situation to the error-analysis implication it supports.
Using Human Performance as a Benchmark for Error Analysis
What should a team expect when its system solves a problem that experienced people cannot judge reliably?
Error Analysis Always Works the Same Way for Every Task
If an ML system depends on abilities that _____ cannot reliably perform, some standard error-analysis techniques stop being useful.
Match each machine learning scenario to its consequence for error analysis tools.
Order the reasoning steps showing how human-performable tasks support better project prioritization.
Using human performance as a reference in machine learning work
When Human Benchmarks Are Unavailable
Why some error analysis methods need a human baseline