Google

If an initial cat-app dev/test set mostly contains adult cat pictures but users later upload many more kitten images than expected, the dev/test set distribution is not representative of the actual distribution the system needs to do well on.

Kitten Uploads Revealing Dev/Test Distribution Mismatch

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Identifying Distribution Mismatch in Cat Apps

Dev/Test Set vs User Uploads

Unrepresentative Dev/Test Set After _____ Uploads

Real-World Distribution Mismatch Elements

Discovering a Distribution Mismatch

Question:
Explain why a cat app evaluated primarily on adult cats might have an unrepresentative dev/test set if users upload many kitten pictures.

Sample answer:
If an initial dev/test set mainly contains pictures of adult cats, the system is being evaluated on its ability to recognize adult cats. However, if users upload a lot more kitten images than expected after the app is shipped, the model encounters a different distribution of data than what it was tested on. This implies that the initial dev/test set distribution is not representative of the actual distribution the system needs to perform well on in the real world.

Key points:
- Dev/test set mainly contained adult cats
- Users uploaded unexpectedly high numbers of kittens
- Dev/test set distribution is not representative
- Mismatch with actual distribution

Rubric:
The response must explain that evaluating on adult cats does not reflect real-world usage if users upload kittens, explicitly concluding that the dev/test set is not representative of the actual distribution.

Addressing Distribution Mismatches in Cat Apps

Case context:
You lead a machine learning team that just launched a cat-spotter app. Your initial dev/test set mainly contained pictures of adult cats. After the launch weekend, you review the logs and find that users are uploading a lot more kitten images than expected.

Question:
Diagnose the primary issue with your evaluation metrics based on this user behavior.

Sample answer:
The primary issue is that the current dev/test set distribution, which is mostly adult cats, is not representative of the actual distribution of images users are uploading (kittens). Therefore, the evaluation process on the initial dev/test set will not accurately reflect how well the system is doing on the actual distribution it needs to do well on.

Key points:
- Initial set mainly adult cats
- Actual user uploads feature many kittens
- Dev/test set is not representative of the actual distribution

Rubric:
Correctly diagnoses that the dev/test set is no longer representative of the actual distribution of user uploads.

The Unexpected Kitten Influx

Question:
What is the consequence if your initial cat app dev/test set had mainly pictures of adult cats, but users later upload many kitten images?

Sample answer:
The consequence is that the dev/test set distribution is not representative of the actual distribution you need the system to do well on.

Key points:
- Dev/test set is not representative
- Actual distribution mismatch

Rubric:
Must state that the dev/test set is unrepresentative of the actual distribution.

Learn Before

Related