Why a Random 30 Percent Split Can Be Misleading When Future Data Will Differ
Question: A team is building a model to predict delivery delays for a grocery service. The data available today comes mostly from one region, but the team expects the model to be deployed in several new regions with different traffic patterns and store behavior. Explain why taking a random 30% of the current data as a test set can be a poor choice. Then describe how the dev and test sets should be selected under these conditions.
Sample answer: A random 30% split works only if the data you hold out looks like the data you expect to face later. Here, the future deployment environment is different from the current data, so a random split would mainly measure performance on the current region instead of the future regions the model must handle. The better approach is to build dev and test sets from data that reflects the future use case as closely as possible. That way, model selection and final evaluation are based on the distribution the system needs to perform well on, not just on what is easiest to sample from the training pool.
Key points:
- A random 30% split can mirror the current data rather than the future deployment data.
- Train, dev, and test sets should not be assumed to come from the same distribution when the future environment differs.
- Dev and test sets should represent the data the model is expected to see after deployment.
Rubric: The answer should explain that a random 30% split is misleading because it evaluates on data similar to the training distribution, and it should state that dev and test sets should be chosen to match the future data distribution the model is intended to serve.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Some Data Should Be Left Out of Training
When Development and Test Sets Reflect Different Populations
How Model Capacity Changes the Risk of Mixing Data Sources
One Predictor Can Work Across Multiple Data Sources
Choose evaluation data to match the real-world target
Mismatched Auxiliary Data Source
Building Dev and Test Sets Before Real Users Exist
Refreshing Evaluation Sets After a Product Launch
Using Public Web Images When No Better Future-Like Data Exists
Judging How Much to Invest in Dev and Test Sets
What should determine dev and test set selection?
True or False: You can assume the training set and test set always come from the same distribution.
Development and test sets should reflect the conditions you expect after deployment, not only the _____ available in your training pool.
Why can a simple random test split be a poor choice when the data you expect in the future is different from the data you have now?
You can usually assume the data used for training and the data used for testing come from the same distribution.
Design Dev and Test Sets for the Future
Match each concept about development and test sets to its description.
Order the steps for choosing development and test sets when future data differs from training data.
What should dev and test examples be designed to resemble?
A validation and test set must exactly match the training distribution in every project.
How should a test set be chosen when deployment data will differ?
Match each data scenario to the best dev/test set choice.
Order the reasoning steps for deciding whether a dev/test split is appropriate.
Why a Random 30 Percent Split Can Be Misleading When Future Data Will Differ
Dev and Test Splits for a Field-Photo Classifier
How to Choose Dev and Test Data When Future Data Will Differ