Google

When several human baselines are available, the best achievable human performance can be used as the human-level proxy for the optimal error rate. In the medical-imaging example, a team of doctors achieves 2% error, so 2% is a reasonable desired performance level and optimal-error proxy; 5% or 10% would be too high because a team can do better.

Use the Best Achievable Human Team to Define Human-Level Performance

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Which human baseline should define the optimal-error proxy for the x-ray diagnosis task?

True or false: The optimal error rate cannot be higher than the doctor team's 2% error.

The reasonable human-level performance proxy is _____ error.

Match each x-ray diagnosis baseline to its observed error rate.

Order the reasoning used to select a human-level performance proxy.

Question:
In a concise analytical response, explain why 2% error should be used instead of 5% or 10% when estimating the optimal error rate and setting desired performance.

Sample answer:
The 2% error achieved by the doctor team is the best demonstrated human performance among the available baselines. Therefore, 5% and 10% are too high as optimal-error proxies because humans can already do better. Using 2% also provides an achievable desired error rate. The true optimal error could be below 2%, but it cannot be above 2% because the team has demonstrated that level.

Key points:
- The doctor team achieves 2% error.
- The 5% and 10% baselines are worse than demonstrated team performance.
- The 2% result is a reasonable optimal-error proxy and desired level.
- The true optimal error may be lower than 2% but cannot be higher.

Rubric:
A strong response identifies the doctor team as the best achievable human baseline, explains why 5% and 10% are too high, connects 2% to both the optimal-error proxy and desired performance, and preserves the distinction between a proxy and the true optimum.

Explain why the doctor team's result is a better proxy than an individual doctor's result.

Case context:
An x-ray diagnosis project has four human baselines: 15% for a typical person with basic training, 10% for a junior doctor, 5% for an experienced doctor, and 2% for a small team that debates each image. Stakeholders propose 5% as the desired error rate.

Question:
Diagnose the problem with the proposed target and recommend the human-level proxy and desired error rate supported by the evidence.

Sample answer:
The 5% proposal uses the experienced individual doctor rather than the best achievable human baseline. Because the doctor team achieves 2% error, 5% is too high as the human-level proxy and desired performance level. The project should use 2% as its reasonable proxy and desired error rate, while allowing that the true optimal error could be lower.

Key points:
- A team outperforms the experienced individual doctor.
- The proposed 5% target is higher than an achievable 2% result.
- Use 2% as the human-level and optimal-error proxy.
- The actual optimal error could be below 2%.

Rubric:
The response should reject 5% using the team evidence, recommend 2% for both the proxy and desired performance, and avoid claiming that 2% must equal the true optimal error.

Choose the appropriate target when stakeholders propose a 5% x-ray diagnosis error rate.

Question:
Answer in one to three sentences using the demonstrated performance of the doctor team.

Sample answer:
The doctor team demonstrates that 2% error is achievable. Therefore, the optimal error rate cannot be higher than 2%, although it could be lower.

Key points:
- The team achieves 2% error.
- Achievable performance means the optimum cannot be higher.
- The optimum could still be lower than 2%.

Rubric:
The answer must connect demonstrated 2% performance to the conclusion that the optimal error cannot exceed 2% and may be lower.

Learn Before

Related