Learn Before
What a test set is for, and what it is not for
Question: In a short analytical response, describe the proper role of a test set and name the kinds of decisions it should not drive.
Sample answer: A test set is meant to estimate how well the final model will perform on unseen data. It should not be used to choose between learning algorithms or to tune hyperparameters. Those decisions should be made using training or validation data, while the test set is reserved for final evaluation.
Key points:
- The test set is for estimating generalization performance.
- It should not determine which learning algorithm is chosen.
- It should not determine hyperparameter settings.
- Model selection and final evaluation serve different purposes.
Rubric: A strong response clearly identifies the test set as an evaluation tool, excludes both algorithm choice and hyperparameter tuning, and separates selection from final assessment.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Machine Learning
Deep Learning
Supervised Learning
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Dev and Test Splits Should Be Smaller for Very Large Datasets
A Test Set Large Enough for a Reliable Final Check
Test Set Size Should Enable Reliable Evaluation
Which use of a test set matches its intended role?
A final test set can measure performance without being used to choose model settings.
The held-out test set is used to _____ the final model’s performance on unseen data.
Match each holdout-set concept with its proper role.
Order the steps for deciding how a test set may be used.
What a test set is for, and what it is not for
Using the Right Evaluation Set for Model Selection
Test Set Purpose and Limits
Using the final holdout set appropriately
The test set should be used to choose the model you deploy.