Question:
When you include some data drawn from the dev/test set distribution in your training set, what hypothesis is further validated if you observe good performance on that specific training data but not on a similar training dev subset?

Sample answer:
It validates the hypothesis that getting more data from that specific target distribution would help improve performance.

Key points:
- Validates the hypothesis that getting more data from the target distribution would help.
- Indicates the model can learn the distribution but is currently overfitting the available examples.

Rubric:
The answer should state that it validates the need for more target-distribution data to improve generalization.

Google

Some data drawn from the same distribution as the dev/test set can be included in the training set. In the in-car audio example, if the training and training dev sets include in-car audio, good performance on in-car training data but not on in-car training dev data further validates the hypothesis that getting more in-car data would help.

Include Some Target-Distribution Examples in Training Alongside Auxiliary Data

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If the training and training dev sets include audio recorded within a car, double-check performance on that subset. If the system does well on car data in the training set but not on car data in the training dev set, that further validates the hypothesis that getting more car data would help.

Checking a Mismatch Hypothesis on Training Dev Subsets

In a cat detector system with 10,000 user-uploaded images and 200,000 internet images, how should you split the target-distribution data?

True or False: We should use target-distribution training data alongside auxiliary data to help the network learn on it.

Adding target-distribution data to training means it contains data from the _____ distribution.

Match the cat detector data components with their distribution properties.

Order the decision process to allocate voice navigation data.

What is a benefit of having in-car audio in both training and training dev sets?

True or False: A training set from a different distribution than the dev/test set should still be used for learning.

Good performance on training but not on training dev validates the _____ hypothesis.

Match the performance outcomes on in-car audio with their implications.

Order the process of setting up and using a cat detector with mixed data.

Question:
In a machine learning project, you have a massive amount of auxiliary data from a different distribution than your dev/test set. Why should you still allocate a portion of your limited target-distribution data to the training set, rather than placing all of it in the dev/test sets? Discuss how this allocation impacts the training process and the evaluation of the model's performance on subsets of the data.

Sample answer:
Including target-distribution data in the training set allows the neural network to directly learn from the specific distribution it will be evaluated on. If you only use auxiliary data for training, the model might struggle to generalize to the target distribution. Additionally, by having target-distribution data in both the training set and a training dev set, you can evaluate performance on the target distribution during training. If the model performs well on target-distribution training data but poorly on target-distribution training dev data, it validates the hypothesis that acquiring more target-distribution data would be beneficial.

Key points:
- Helps the neural network learn directly from the target distribution.
- Enables evaluation of performance specifically on the target distribution during training.
- Comparing performance on target-distribution training data vs. target-distribution training dev data helps determine if more target-distribution data is needed.

Rubric:
The response should explain that target-distribution data helps the model learn the specific characteristics of the target domain. It should also discuss how having this data in both training and training dev sets allows for specific performance comparisons to validate hypotheses about data requirements.

Explain the rationale for including target-distribution examples in the training set.

Case context:
You are building a speech recognition system for a voice-controlled mobile navigation app. You have 500,000 general audio clips (auxiliary data) and 20,000 clips of users speaking street addresses (target distribution). You decide to use 10,000 street address clips for the dev/test sets. The remaining 10,000 street address clips are combined with the 500,000 general clips for training. During training, you monitor performance specifically on the street address data within your training set and a separate training dev set of street addresses.

Question:
If your system achieves high accuracy on the 10,000 street address examples in the training set, but poor accuracy on the street address examples in your training dev set, what should you diagnose as the primary issue, and what actionable decision should you make based on this validation?

Sample answer:
The high accuracy on the street address training data combined with poor accuracy on the street address training dev data indicates that the model is overfitting specifically to the limited target-distribution data it was trained on. This outcome validates the hypothesis that the model needs more data from the target distribution to generalize well. The actionable decision is to focus efforts on acquiring more street address audio clips to expand the target-distribution training data.

Key points:
- Diagnose overfitting to the available target-distribution training examples.
- Validate the hypothesis that more target-distribution data is necessary.
- Decide to acquire more data matching the dev/test distribution (street addresses).

Rubric:
The response must diagnose the issue as overfitting to the available target-distribution data and decide that the team should acquire more data from that specific distribution.

Learn Before

Related