Question: Explain why error analysis processes are more powerful when an ML system automates a human-solvable problem. How does this impact a development team's ability to prioritize their work efficiently?

Sample answer: When an ML system automates a task that humans can do, developers can use human-level performance as a benchmark to evaluate the system. This benchmarking allows the team to pinpoint exactly where the system is falling short compared to a human. With these powerful error analysis tools, the team can clearly identify which components or specific errors are causing the most significant performance gaps, thus allowing them to prioritize their work efficiently on the most impactful areas.

Key points:

• Automating human-solvable tasks allows benchmarking against human-level performance.
• Human-level benchmarking provides a clear reference point for evaluating system output.
• This results in more powerful and applicable error analysis tools.
• Better error analysis allows teams to identify the most critical areas for improvement.
• Consequently, teams can prioritize their development work more efficiently.

Rubric: The essay should clearly connect human-solvable problems to the availability of human-level performance benchmarks. It must explain how these benchmarks serve as powerful error analysis tools and conclude by linking these tools to the team's ability to efficiently prioritize their work.