Building Dev and Test Sets Before Real Users Exist
Before a product is released, there may be no genuine user data yet for development and test evaluation. In that situation, the best approach is to build proxy sets that resemble the data expected after launch. For example, if a plant-identification app will be used outdoors on phones, the team could collect photos from volunteers using varied lighting, distances, and camera quality rather than relying on polished studio images. The goal is to approximate the future distribution as closely as possible, even though it will not be exact.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Some Data Should Be Left Out of Training
When Development and Test Sets Reflect Different Populations
How Model Capacity Changes the Risk of Mixing Data Sources
One Predictor Can Work Across Multiple Data Sources
Choose evaluation data to match the real-world target
Mismatched Auxiliary Data Source
Building Dev and Test Sets Before Real Users Exist
Refreshing Evaluation Sets After a Product Launch
Using Public Web Images When No Better Future-Like Data Exists
Judging How Much to Invest in Dev and Test Sets
What should determine dev and test set selection?
True or False: You can assume the training set and test set always come from the same distribution.
Development and test sets should reflect the conditions you expect after deployment, not only the _____ available in your training pool.
Why can a simple random test split be a poor choice when the data you expect in the future is different from the data you have now?
You can usually assume the data used for training and the data used for testing come from the same distribution.
Design Dev and Test Sets for the Future
Match each concept about development and test sets to its description.
Order the steps for choosing development and test sets when future data differs from training data.
What should dev and test examples be designed to resemble?
A validation and test set must exactly match the training distribution in every project.
How should a test set be chosen when deployment data will differ?
Match each data scenario to the best dev/test set choice.
Order the reasoning steps for deciding whether a dev/test split is appropriate.
Why a Random 30 Percent Split Can Be Misleading When Future Data Will Differ
Dev and Test Splits for a Field-Photo Classifier
How to Choose Dev and Test Data When Future Data Will Differ
Learn After
Before a new language-learning app launches, how can the team estimate future dev/test data if no real user logs exist yet?
Before a mobile app launches, the exact future data distribution for dev and test sets can be known in advance.
Before launching a receipt-sorting app, what kind of sample photos should you collect so they resemble the images real users will upload later?
How should dev/test sets be created before a mobile app launches if no real user data exists yet?
Before a subscription streaming service launches, it is impossible to assemble a development or test set because no customer viewing logs exist yet.
Before launch, if no real user data exists, try to _____ the future distribution for dev/test.
Match each pre-launch data idea to the description it fits.
Order the steps for preparing a stand-in dev/test set before a product launches.
Which data collection approach best approximates the kinds of images a mobile plant-identification app will receive after launch?
Prelaunch survey data collected from a small group of acquaintances will exactly match the distribution of future users.
Ask your classmates for proxy user photos
Match each pre-launch data approximation term to its correct explanation.
Order the steps for handling missing future-user data when preparing evaluation sets before a delivery app launches.
Why approximate future dev/test data before a product launch?
Planning a prelaunch evaluation set for a pet-recognition app
Why Gather Sample User Images Before Launch?