Learn Before
Test Set Purpose and Limits
Question: Answer in one to three sentences.
Sample answer: A test set is kept aside so it can give a final estimate of how well a trained model will generalize to new data. It should not be used to choose among candidate models or to adjust hyperparameters.
Key points:
- Provides a final estimate of generalization.
- Do not use it to select the model.
- Do not use it to tune hyperparameters.
Rubric: The answer should describe the final evaluation purpose and reject both model selection and hyperparameter tuning.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Machine Learning
Deep Learning
Supervised Learning
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Dev and Test Splits Should Be Smaller for Very Large Datasets
A Test Set Large Enough for a Reliable Final Check
Test Set Size Should Enable Reliable Evaluation
Which use of a test set matches its intended role?
A final test set can measure performance without being used to choose model settings.
The held-out test set is used to _____ the final model’s performance on unseen data.
Match each holdout-set concept with its proper role.
Order the steps for deciding how a test set may be used.
What a test set is for, and what it is not for
Using the Right Evaluation Set for Model Selection
Test Set Purpose and Limits
Using the final holdout set appropriately
The test set should be used to choose the model you deploy.