Google

Data mismatch occurs when an algorithm generalizes well to new data from the training-set distribution but not to data from the dev/test distribution. It is because the training set data is a poor match for the dev/test set data.

Data Mismatch Between Training and Dev Set Distributions

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A data mismatch problem can appear when humans achieve near perfect performance, the algorithm has 1% training error, 1.5% error on unseen data from the training-set distribution, and 10% dev-set error.

Example Data Mismatch Error Pattern

When there is a data mismatch problem, one recommended option is to find more training data that better matches the dev-set examples the algorithm has trouble with. In the speech-recognition example, when the dev set mostly contains in-car audio while training examples were mostly recorded against a quiet background, one might try to acquire more training data made of audio clips taken in a car.

Finding Training Data That Better Matches Difficult Dev Examples

When a data mismatch problem is found, a recommended step is to try to understand which properties of the data differ between the training and dev-set distributions.

Addressing Data Mismatch by Comparing Data Properties

What defines data mismatch in the context of training and dev/test sets?

Data mismatch means the algorithm fails to generalize even to new data drawn from the training distribution.

Data mismatch is named because the training set data is a poor _____ for the dev/test set data.

Match each data mismatch term to its correct description.

Order the steps for diagnosing a data mismatch problem from training through evaluation.

What is the root cause of a data mismatch problem between training and dev/test sets?

A speech recognition system that does well on training and training dev sets but poorly on the dev set has a data mismatch problem.

An algorithm with data mismatch generalizes well to the _____ distribution but not to the dev/test distribution.

Match each performance pattern to its diagnostic interpretation regarding data mismatch.

Order the reasoning chain that leads from observation to a data mismatch conclusion.

Question:
Based on the provided concepts, explain why an algorithm can generalize well to new data from the training-set distribution but fail on the dev/test set distribution. What does this performance gap indicate about the training data?

Sample answer:
This gap indicates a data mismatch problem. The algorithm generalizes well to new data from the training-set distribution because it has learned that distribution. However, it fails to generalize to the dev/test set distribution because the training set data is a poor match for the dev/test set data.

Key points:
- The algorithm generalizes well to new data from the training-set distribution.
- The algorithm does not generalize well to data from the dev/test set distribution.
- This performance gap is called data mismatch.
- The underlying cause is that the training set data is a poor match for the dev/test set data.

Rubric:
The answer should explain that the algorithm generalizes well to the training distribution but not the dev/test distribution because of data mismatch, which is caused by the training set data being a poor match for the dev/test set data.

Explain how generalization performance differences between training and dev/test distributions indicate data mismatch.

Case context:
Suppose you have developed a speech recognition system that does very well on the training set and on the training dev set. However, it does poorly on your dev set.

Question:
Based on this scenario, diagnose the specific problem your speech recognition system is experiencing and explain the underlying reason for this issue.

Sample answer:
The speech recognition system is experiencing a data mismatch problem. This occurs because the training set data is a poor match for the dev set data, causing the system to perform well on the training and training dev sets (which share the same distribution) but poorly on the dev set.

Key points:
- Diagnose the issue as a data mismatch problem.
- Acknowledge the system performs well on the training set and training dev set but poorly on the dev set.
- Explain that the training set data is a poor match for the dev set data.

Rubric:
The student must diagnose the problem as data mismatch and explain that it is caused by the training set data being a poor match for the dev set data.

Diagnose generalization issues in a speech recognition system.

Question:
Why is the term 'data mismatch' specifically chosen to describe the problem where an algorithm generalizes well to the training distribution but poorly to the dev/test distribution?

Sample answer:
The problem is called data mismatch because the training set data is a poor match for the dev/test set data.

Key points:
- The training set data is a poor match for the dev/test set data.

Rubric:
The response must state that the name arises because the training set data is a poor match for the dev/test set data.

Learn Before

Related