Progress Slows After Machines Surpass Human-Level Performance
Once humans have a hard time identifying examples that an algorithm is clearly getting wrong, only a subset of human-comparison techniques still apply. Progress is therefore usually slower on problems where machines already surpass human-level performance, while it is faster when machines are still catching up to humans.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
Progress Slows After Machines Surpass Human-Level Performance
When does comparing to human performance still help an ML system that already surpasses average human-level accuracy on the dev/test set?
Even after a system surpasses average human-level performance on the full dev/test set, human comparison can still provide value on specific data subsets.
On subsets where humans outperform the algorithm, humans can still provide better _____, useful intuition, and a desired performance target.
Match each benefit of using human comparison on a human-better data subset to its description.
Order the reasoning steps for deciding whether human-comparison techniques still apply after your system surpasses average human-level performance.
In the MLY speech recognition example, at which task does the system surpass humans while humans still outperform the system at a different task?
Once a system's average performance on the full dev set exceeds human-level performance, human-comparison techniques like error analysis and human labeling no longer apply at all.
According to MLY, human-comparison techniques apply 'so long as there are dev set examples where humans are _____ and your algorithm is wrong.'
Match each element of the MLY speech recognition example to its role in the human-better-subset framework.
Order the steps for leveraging a human-better subset—like rapidly spoken speech in the MLY example—to improve an ML system.