Link between human-solvable problems and error analysis tools
Question: Briefly explain why some error analysis procedures do not apply to ML systems where intermediate components perform tasks humans cannot do well.
Sample answer: Error analysis procedures often rely on human-level performance as a benchmark. If intermediate components perform tasks humans cannot do well, this benchmark is unavailable, rendering those specific error analysis procedures inapplicable.
Key points:
- Error analysis often benchmarks against human-level performance.
- Tasks humans cannot do well lack this human benchmark.
Rubric: The answer should mention the reliance on human-level benchmarks and state that the inability of humans to perform the task removes the benchmark needed for certain error analysis procedures.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why do error analysis processes work best when an ML system automates a human-solvable task?
When an ML system automates a task humans can do well, human-level performance can serve as a benchmark for error analysis.
Working on human-solvable problems provides more powerful _____ tools, enabling more efficient team prioritization.
Match each task characteristic to its implication for applying error analysis procedures.
Order the steps for applying human-level benchmarking in error analysis for a human-solvable ML task.
An ML team builds a system performing a task no human expert can reliably evaluate. What should the team expect about error analysis?
All standard error analysis procedures in Machine Learning Yearning apply equally regardless of whether humans can perform the task.
If an ML system's final output or intermediate components do things _____ cannot do well, some error analysis procedures will not apply.
Match each ML development scenario to its consequence for error analysis tooling.
Order the reasoning steps explaining why human-solvable problems lead to more efficient team prioritization in ML development.
Assess the value of human-level benchmarks in prioritizing ML team tasks
Diagnosing error analysis limitations in a novel ML application
Link between human-solvable problems and error analysis tools