Case Study

Set the performance target using the strongest human benchmark, not a weaker individual baseline.

Scenario: A plant-quality team is building a system to detect faulty circuit boards. Four human reference points are available: 18% error for a new trainee, 11% for a line operator with moderate experience, 7% for a senior inspector, and 3% for a two-person review team that examines each board together. Management suggests that 7% should be the goal.

Question: Identify the problem with the proposed goal and state the human-level proxy and target error rate that the evidence supports.

Sample answer: The 7% goal is based on the senior individual inspector, but the strongest available human comparison is the two-person review team at 3% error. That means 7% is too loose if the team can already do better. The project should use 3% as the human-level proxy and as the practical target for the system, while recognizing that the true achievable minimum could be even lower.

Key points:

  • The review team performs better than the best single inspector.
  • A 7% target is worse than an already demonstrated 3% result.
  • Use 3% as the human-level proxy and working performance target.
  • The true optimal error rate may be below 3%.

Rubric: The response should reject 7% by citing the two-person team, recommend 3% for both the proxy and the desired target, and avoid claiming that 3% must equal the true minimum error.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI