Google

When an algorithm is worse than humans on a task, error analysis can use human intuition to understand what information a person used to get the right answer and then use that knowledge to modify the learning algorithm. In medical imaging, discussing images with doctors can similarly expose useful intuitions.

Human Intuition Can Guide Error Analysis

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When is human-intuition-guided error analysis most applicable in ML development?

The goal of drawing on human intuition in error analysis is to understand what information a person uses to get the correct answer.

In the speech recognition example, the algorithm incorrectly transcribed 'pair of apples' as '_____ of apples.'

Match each element of human-intuition-guided error analysis to its role in the process.

Order the steps for conducting human-intuition-guided error analysis on an underperforming algorithm.

In medical imaging, how does Machine Learning Yearning recommend drawing on human intuition during error analysis?

Human-intuition error analysis is equally useful whether the algorithm outperforms or underperforms humans.

In Machine Learning Yearning, drawing on human _____ during error analysis helps identify what information leads to the correct answer.

Match each domain or scenario to the correct description of how human intuition is applied in error analysis.

Order the steps a team follows when using doctor intuitions to improve a medical imaging algorithm.

Question:
Discuss how human intuition can be used during error analysis when a machine learning algorithm's performance is below human-level, using the speech recognition example from Machine Learning Yearning to illustrate your point.

Sample answer:
When an algorithm performs worse than humans, developers can use error analysis to understand what specific information a person uses to arrive at the correct answer. For example, if a speech recognition algorithm confuses "pair" and "pear" in "a pair of apples," developers can rely on human intuition about context (knowing apples come in pairs) to understand the failure. This knowledge is then used to modify the learning algorithm so it can capture the same contextual clues that humans use naturally.

Key points:
- Algorithm is worse than human-level
- Understand information humans use to get the right answer
- Modify the learning algorithm
- Speech recognition example involving context of 'pair of apples'

Rubric:
A good answer will identify the condition (algorithm is worse than human level), explain the mechanism (understanding what information humans use), and apply the speech recognition example correctly.

Explain how human intuition improves underperforming ML models.

Case context:
Your team has deployed a medical imaging algorithm to detect a specific abnormality. However, during testing, the algorithm performs worse than the team of human doctors on the task.

Question:
According to Machine Learning Yearning, what is the recommended approach to diagnose and improve this algorithm using the doctors' expertise, and what exactly are you trying to extract from them?

Sample answer:
The recommended approach is to discuss the specific misclassified images with the team of doctors to draw on their intuitions. The goal is to understand exactly what visual information or context the doctors are using to arrive at the correct diagnosis, so that this knowledge can be used to modify the learning algorithm to look for similar information.

Key points:
- Discuss images with a team of doctors
- Draw on doctors' intuitions
- Understand what information they use to get the correct answer
- Use the knowledge to modify the algorithm

Rubric:
A good answer should mention discussing the images directly with doctors and the goal of extracting the specific information/intuitions they use to make correct decisions.

Diagnosing a Medical Imaging Algorithm using Doctor Intuitions

Question:
Briefly explain the primary goal of drawing on human intuition when conducting error analysis on an algorithm that performs worse than humans.

Sample answer:
The primary goal is to understand exactly what information a human uses to get the correct answer, and then use that understanding to modify and improve the learning algorithm.

Key points:
- Understand what information a person uses
- Modify the learning algorithm

Rubric:
The answer must state that the goal is to understand the information humans use and to apply that knowledge to modify the algorithm.

Learn Before

Related