Question:
When evaluating a single pipeline component (e.g., 'Plan path' in a self-driving system) against human-level performance, what specific restriction must be placed on the information given to the human?

Sample answer:
To properly evaluate the component, the human must be given only the exact same inputs that the specific pipeline component receives (such as the outputs from the previous components), rather than having access to the original raw data like camera images.

Key points:
- The human must receive the exact same input as the algorithm component.
- The human should be restricted to the previous components' outputs, not the raw data.

Rubric:
The answer must specify that the human evaluator is restricted to using only the outputs from the preceding pipeline components, ensuring a fair comparison.

Google

In cat recognition, because a human can recognize whether a picture contains a cat almost all the time, the ideal error rate achievable by an optimal classifier is nearly 0%. In a speech-recognition task where 14% of audio clips are too noisy or unintelligible for even a human to recognize, even the most optimal speech recognition system might have error around 14%.

Human-Level Performance as a Proxy for Optimal Error Rate

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When several human baselines are available, the best achievable human performance can be used as the human-level proxy for the optimal error rate. In the medical-imaging example, a team of doctors achieves 2% error, so 2% is a reasonable desired performance level and optimal-error proxy; 5% or 10% would be too high because a team can do better.

Use the Best Achievable Human Team to Define Human-Level Performance

Human-level performance on mobile images can be measured by asking humans to label the mobile cat-image data and measuring their error.

Measuring Human Error on Mobile-Image Examples

When humans achieve almost perfect cat-detection performance and the optimal error rate is about 0%, an algorithm with 1% training error, 5% training-dev error, and 5% dev error has high variance.

High Variance Indicated by 1% Training Error and 5% Training-Dev Error

Why is the optimal error rate for cat recognition nearly 0% according to Machine Learning Yearning?

If 14% of audio clips are too noisy for humans to understand, the optimal speech recognition error rate is approximately 14%.

Human-level performance is used as a proxy to estimate the _____ error rate on a given task.

Match each task scenario to the approximate optimal error rate implied by human-level performance.

Order the steps for using human-level performance to estimate optimal error rate and guide bias reduction.

An algorithm achieves 10% error on a task where humans achieve 2% error. What is the avoidable bias and what action does this suggest?

Human-level performance always equals 0% error, so the optimal error rate is always 0% for any machine learning task.

In the cat recognition example, because a human can recognize cats almost all the time, the ideal error rate is nearly _____.

Match each key term to its definition in the context of human-level performance as an optimal error rate proxy.

Order the reasoning steps for deciding whether a task's optimal error rate is near 0% or substantially higher.

Question:
Using a specific example, explain how the gap between an algorithm's error and human-level performance helps estimate avoidable bias and directs developers toward bias-reducing techniques.

Sample answer:
By comparing an algorithm's error rate to human-level performance (which acts as a proxy for the optimal error rate), developers can estimate avoidable bias. For example, if an algorithm achieves a 10% error rate but humans achieve 2%, the avoidable bias is at least 8%. This large gap indicates that the algorithm is underfitting the training data, meaning developers should focus on bias-reducing techniques such as training a larger model or training for longer.

Key points:
- Human-level performance acts as a proxy for optimal error rate.
- The gap between algorithm error and human error estimates avoidable bias.
- High avoidable bias indicates the need for bias-reducing techniques.

Rubric:
A correct response will identify human performance as a proxy for optimal error, provide an example calculating avoidable bias (e.g., 10% algorithm error minus 2% human error equals 8% avoidable bias), and state that a high avoidable bias requires bias-reducing techniques.

Explain how comparing algorithm and human-level performance guides bias reduction.

Case context:
A team is building a speech-recognition system for a factory. Upon review, they find that 14% of the collected audio clips are so distorted by machinery noise that even human experts cannot transcribe them accurately. The team's current algorithm achieves a 16% error rate.

Question:
Based on the provided case context, what should the engineering team set as their desired error rate proxy, and how does this impact their goal setting for the algorithm?

Sample answer:
The team should set their desired optimal error rate proxy around 14%, because human experts cannot surpass this threshold due to the inherent noise in the data. Since the current algorithm is at a 16% error rate, the avoidable bias is only about 2%. Therefore, the team should recognize that a near 0% error rate is impossible without fixing the input data, and they should adjust their goals to focus on closing the remaining 2% gap or improving data quality.

Key points:
- The optimal error rate proxy is 14% due to the unintelligible audio.
- The algorithm's avoidable bias compared to human-level performance is roughly 2%.
- A near 0% error rate is an unrealistic goal for this specific dataset.

Rubric:
A strong response will identify 14% as the proxy for the optimal error rate, calculate the avoidable bias as 2%, and conclude that pushing for a 0% error rate on this dataset is an unrealistic goal.

Learn Before

Related