What the dev/test set says about deployment data
A team builds a classifier for factory inspection photos. The dev and test sets are drawn from a fixed overhead camera inside a brightly lit packing line. After launch, however, most new inputs come from handheld phone photos uploaded by field inspectors in warehouses with uneven lighting. What does this tell you about the dev/test set?
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Recognizing a Dev/Test Distribution Shift
A video app can keep using a test set made only of desktop screen recordings after most users switch to short phone clips.
A Dev/Test Set Should Match _____ Data
Distribution Shift in a Wildlife Classifier
Finding a Dataset Distribution Mismatch
Why a Test Set Can Fail to Reflect Real Usage
The Shift in Field Photos
Validation Data and Production Shift
What the dev/test set says about deployment data
When the Evaluation Set No Longer Matches User Behavior