Google

Once dev and test sets are defined, a team will focus on improving dev set performance, so the dev set should reflect the task one wants to improve on the most. Having different dev and test set distributions can lead to a system that works well on the dev set but does poorly on the test set. If the dev and test sets came from the same distribution, that result would have a clear diagnosis: the system has overfit the dev set, and the obvious cure is to get more dev set data. If the dev and test sets come from different distributions, the options are less clear: the system may have overfit the dev set, the test set may be harder than the dev set, or the algorithm might be doing as well as could be expected.

Choosing Dev and Test Sets from the Same Distribution When Possible

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Because a team focuses on improving dev set performance after defining dev and test sets, the dev set should reflect the task the team most wants to improve on.

Dev Set Should Reflect the Task to Improve Most

If a system works well on the dev set but not the test set and the dev and test sets come from the same distribution, the clear diagnosis is that the system has overfit the dev set; the obvious cure is to get more dev set data.

Same-Distribution Dev/Test Failure Indicates Dev Set Overfitting

If dev and test sets come from different distributions, a system that works well on the dev set but poorly on the test set has several possible explanations: dev-set overfitting, a harder test set, or a test set that is different rather than harder.

Different Dev/Test Distributions Make Failure Diagnosis Ambiguous

On a third-party benchmark whose creator specified dev and test sets from different distributions, luck can have a greater impact on performance than it would if the dev and test sets came from the same distribution.

Third-Party Benchmark Distribution Mismatch Increases Luck

When dev and test sets share the same distribution and test performance is worse than dev performance, what does this clearly indicate?

True or False: When dev and test sets come from different distributions, a performance gap between them has a single, unambiguous diagnosis.

When a system has overfit the dev set and both sets share the same distribution, the obvious cure is to get more _____ data.

Why should the dev set reflect the task a team wants to improve on the most?

If both sets share the same distribution and a model performs well on dev but poorly on test, the clear diagnosis is dev set overfitting.

When a model overfits the dev set and both sets share the same distribution, the obvious cure is to get more _____ data.

Match each dev/test set scenario to its consequence for model diagnosis.

Order the diagnostic steps when a model works well on the dev set but fails on the test set.

Which is a possible explanation for poor test performance when dev and test sets come from different distributions?

When dev and test sets come from different distributions, a system's failure on the test set provides an unambiguous diagnosis.

Once the dev and test sets are defined, a team will be focused on improving _____ set performance.

Match each concept related to dev/test distribution to its correct description.

Order the steps for selecting dev and test sets that support clear model evaluation.

Question:
Suppose a model performs well on the dev set but poorly on the test set. Analyze how the diagnosis of this issue and the options for fixing it differ when the dev and test sets come from the same distribution versus when they come from different distributions.

Sample answer:
If the dev and test sets come from the same distribution, the diagnosis is clear: the system has overfit the dev set, and the obvious cure is to obtain more dev set data. If the dev and test sets come from different distributions, the diagnosis is ambiguous and options are less clear. The failure could be because the system overfit the dev set, the test set is harder than the dev set, or the algorithm is doing as well as could be expected.

Key points:
- Under the same distribution, poor test set performance clearly indicates the model has overfit the dev set.
- The obvious cure for dev set overfitting when distributions match is to collect more dev set data.
- If distributions differ, diagnosing poor test performance becomes ambiguous and options are less clear.
- Potential reasons for failure under different distributions include dev set overfitting, a harder test set, or the algorithm doing as well as expected.

Rubric:
To receive full credit, the answer must identify that: 1. Under the same distribution, the diagnosis is overfitting the dev set and the cure is getting more dev set data. 2. Under different distributions, the diagnosis is ambiguous. 3. The potential causes under different distributions include: overfitting the dev set, the test set being harder, or the algorithm doing as well as could be expected.

Compare the diagnostics of poor test performance under same vs. different dev/test distributions.

Case context:
A team builds a speech recognition system. They optimize their model's parameters and achieve excellent performance on their development set. However, when they evaluate the final model on their test set, they observe a significant drop in accuracy. They then check their data sources and find that the dev set and test set were collected from different distributions.

Question:
Given that the dev and test sets come from different distributions, explain why the team cannot make a definitive diagnosis about why the model failed on the test set, and list the possible explanations they must consider.

Sample answer:
Because the dev and test sets come from different distributions, the diagnosis of the poor test set performance is ambiguous. The team cannot definitively isolate the issue. They must consider three possibilities: they may have overfit the dev set, the test set may simply be harder than the dev set, or their algorithm might be performing as well as could be expected.

Key points:
- Different dev and test distributions make diagnosing poor test performance ambiguous.
- The team may have overfit the model to the dev set.
- The test set might be harder than the dev set.
- The algorithm might be doing as well as could be expected.

Rubric:
The answer should explain that different distributions make the diagnosis ambiguous or unclear. It must list the three possible explanations grounded in the source text: 1) Overfitting the dev set, 2) The test set being harder than the dev set, and 3) The algorithm doing as well as could be expected.

Diagnosing a drop in test set performance with mismatched distributions.

Question:
If a machine learning system performs well on the dev set but poorly on the test set, and both sets come from the exact same distribution, what is the clear diagnosis of the problem and its obvious cure?

Sample answer:
The clear diagnosis is that the system has overfit the dev set. The obvious cure for this issue is to get more dev set data.

Key points:
- The diagnosis is that the model has overfit the dev set.
- The cure is to obtain more dev set data.

Rubric:
The response must identify 'overfitting the dev set' (or equivalent phrase) as the diagnosis and 'getting more dev set data' (or equivalent phrase) as the cure.

Learn Before

Related