Diagnose the evaluation-set mistake in a mobile cat-classification app.
Case context: A team trains a cat classifier using many downloaded internet images. Its dev and test sets also contain website images, but app users submit lower-resolution, blurrier, and poorly lit mobile-phone pictures. The classifier performs poorly on those uploads.
Question: What should the team diagnose, and how should it revise its dev and test sets?
Sample answer: The team should diagnose a mismatch between its website-image evaluation distribution and the mobile-image distribution it actually cares about. It should construct dev and test sets from user-uploaded mobile images because those reflect expected future inputs and the distribution on which the app should perform well. Internet images may still be used as additional training data.
Key points:
- Website images differ from mobile uploads.
- Current evaluation does not represent the target distribution.
- Dev and test sets should use mobile images.
- Internet images may remain useful for training.
Rubric: The response should identify the distribution mismatch, select mobile images for dev/test, justify that choice using future target performance, and avoid claiming that internet images must be discarded from training.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Avoid Randomly Shuffling Mixed-Source Data into Dev/Test Sets
Include Some Target-Distribution Examples in Training Alongside Auxiliary Data
Down-Weighting Auxiliary Data from a Different Distribution
Training Dev Set
Error Table Across Two Data Distributions and Three Error Types
Data Mismatch Between Training and Dev Set Distributions
Limited Practical Scope of Domain Adaptation for Different Data Distributions
Domain Adaptation for Different Data Distributions
Website Images and Mobile Phone Pictures as a Distribution Mismatch Example
Random 70/30 Train/Test Split Can Fail Under Distribution Shift
Which data should define the dev and test sets for the cat-picture app?
Dev and test sets should represent the future data distribution of interest.
Complete the principle: Dev and test sets should reflect _____ data.
Match each cat-app data group with its appropriate role or property.
Order the decisions for building datasets when auxiliary and target data differ.
Explain why different training and evaluation distributions can be appropriate.
Diagnose the evaluation-set mistake in a mobile cat-classification app.
Why did success on website images fail to ensure success on mobile uploads?
Which dataset design best uses both target and auxiliary cat images?
Using extra internet images for training requires internet images in dev and test sets.