Google

Human-level performance on mobile images can be measured by asking humans to label the mobile cat-image data and measuring their error.

Measuring Human Error on Mobile-Image Examples

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Determining Mobile Image Human Performance

Labeling Data for Mobile Image Performance

Human-level performance on mobile images can be measured by asking humans to _____ the mobile cat-image data and measuring their error.

Components of Mobile Image Evaluation

Steps to Establish Mobile Human Performance

Question:
Explain the methodology for establishing a human-level performance benchmark for a specific data subset, such as mobile cat images.

Sample answer:
To establish a human-level performance benchmark for a specific data subset like mobile images, you cannot rely on human performance on a different distribution (like internet images). Instead, you must gather examples specifically from the target distribution (mobile cat images). You then ask humans to perform the task—in this case, labeling the images. Finally, you measure the error rate of the human labelers on this specific dataset. This resulting error rate represents the human-level performance for that exact data distribution.

Key points:
- Use data from the specific target distribution (e.g., mobile images).
- Ask humans to label this specific data.
- Measure the resulting human error rate on the target data.

Rubric:
A strong answer will explicitly mention using the specific target data distribution, having humans perform the labeling task on that data, and calculating their resulting error rate.

Methodology for Mobile Benchmark

Case context:
Your team has established human-level performance for high-resolution web images at 1%. However, your application will deploy on mobile phones, where images are blurry and poorly lit. You are trying to determine if your algorithm's 5% error on mobile data is acceptable.

Question:
What specific action must your team take to determine an appropriate benchmark for the mobile application data?

Sample answer:
The team must take a sample of the actual blurry, poorly lit mobile images and ask humans to label them. By measuring the human error rate on this specific mobile dataset, they can establish the human-level performance benchmark for the mobile distribution.

Key points:
- Use the mobile image dataset.
- Have humans label the mobile images.
- Measure the human error rate to establish the benchmark.

Rubric:
The response must identify that humans need to label the mobile data specifically to measure their error rate on that distribution.

Diagnosing Missing Benchmarks

Question:
What specific action must humans take with mobile cat-image data in order to measure their performance level?

Sample answer:
Humans must label the mobile cat-image data.

Key points:
- Humans must label the data.

Rubric:
Award full credit for stating that humans must label or categorize the data.

Learn Before

Related