Laboratory Photos and Factory Tablet Images as a Distribution Mismatch Example
A parts-inspection classifier trained and evaluated on clean laboratory photos from a fixed camera can perform worse after deployment on factory-floor tablet images. The tablet photos are often lower-resolution, blurred by hand movement, and unevenly lit, so the original evaluation set does not represent the deployment distribution.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Do Not Randomly Mix Sources When Building Validation and Test Sets
Use a Small Amount of Target-Distribution Data in Training with Auxiliary Data
When to Reduce the Influence of Extra Training Data
Training-Development Set
Training–Dev Distribution Gap
Choosing Development and Test Data for a Specific Application
Random Train/Test Splits Can Be Misleading When Deployment Data Differs
Which data should define the dev and test sets for the plant-disease app?
Development and test examples should be drawn to match the kind of data the deployed system is expected to see.
Development and test sets should match the kind of data you expect later.
Match each data group for the plant-disease app with its role.
Order the dataset choices when training data and evaluation data come from different sources.
Why training data and evaluation data may come from different sources
Find the evaluation-set mismatch in a voice-command detector.
Why did strong desktop-photo results not predict mobile-upload success?
Choosing Training and Evaluation Data from Two Image Sources
If training uses extra web-sourced photos, the dev and test sets must also contain web-sourced photos.
Comparing Errors Across Two Data Sources
Laboratory Photos and Factory Tablet Images as a Distribution Mismatch Example
Learn After
Why did the deployed flower classifier work well on its original test set but poorly after launch?
The evaluation set matched the camera conditions seen during deployment.
Field Photos from Visitor Tablets
Match each part of the bird-photo classifier example to its role in the distribution shift.
Order the steps that exposed a data-distribution mismatch in an email spam filter.
Why Did Good Results on Scanned Documents Not Guarantee Success on Phone Photos?
Diagnose a document classifier that works on office scans but fails on phone photos.
How User Photos Differed from Catalog Images
What did the model's strong result on the website-based test set actually show?
If a model is trained and tested only on studio photos, that evaluation can still miss poor performance on blurry customer-uploaded phone pictures.