Google

A test set is used to estimate how well a trained system performs, but it should not be used to choose the model, features, or hyperparameters.

Purpose of a Test Set

A large holdout split is useful when data is limited, and an older rule of thumb was to reserve about 30% of the data for testing. That kind of split can be reasonable for small to medium datasets, such as those with roughly 100 to 10,000 examples. When datasets reach much larger scales, it is usually better to allocate a much smaller percentage to dev and test sets. The key requirement is that these sets be large enough to give a dependable estimate of performance, but no larger than needed for that purpose.

Dev and Test Splits Should Be Smaller for Very Large Datasets

A test set should contain enough examples to give a reliable estimate of the system's final performance.

A Test Set Large Enough for a Reliable Final Check

The test set should contain enough examples to make the estimate of overall system performance reliable.

Test Set Size Should Enable Reliable Evaluation

Which use of a test set matches its intended role?

A final test set can measure performance without being used to choose model settings.

The held-out test set is used to _____ the final model’s performance on unseen data.

Match each holdout-set concept with its proper role.

Order the steps for deciding how a test set may be used.

Question:
In a short analytical response, describe the proper role of a test set and name the kinds of decisions it should not drive.

Sample answer:
A test set is meant to estimate how well the final model will perform on unseen data. It should not be used to choose between learning algorithms or to tune hyperparameters. Those decisions should be made using training or validation data, while the test set is reserved for final evaluation.

Key points:
- The test set is for estimating generalization performance.
- It should not determine which learning algorithm is chosen.
- It should not determine hyperparameter settings.
- Model selection and final evaluation serve different purposes.

Rubric:
A strong response clearly identifies the test set as an evaluation tool, excludes both algorithm choice and hyperparameter tuning, and separates selection from final assessment.

What a test set is for, and what it is not for

Case context:
A data science team trains several models, compares their results on a held-out test set, and then uses those same test results to choose the final model and tune its hyperparameters.

Question:
What is wrong with this workflow, and what should the test set be used for instead?

Sample answer:
The team is using the test set for decisions it should not influence. Model selection and hyperparameter tuning should be based on training or validation data, while the test set should be saved for a final, unbiased evaluation after all choices are finished.

Key points:
- Do not use the test set to choose among models.
- Do not use the test set to tune parameters.
- Use the test set only for final performance evaluation.

Rubric:
The response must identify both improper uses of the test set—model selection and parameter tuning—and explain that the test set should be reserved for final evaluation.

Using the Right Evaluation Set for Model Selection

Question:
Answer in one to three sentences.

Sample answer:
A test set is kept aside so it can give a final estimate of how well a trained model will generalize to new data. It should not be used to choose among candidate models or to adjust hyperparameters.

Key points:
- Provides a final estimate of generalization.
- Do not use it to select the model.
- Do not use it to tune hyperparameters.

Rubric:
The answer should describe the final evaluation purpose and reject both model selection and hyperparameter tuning.

Learn Before

Related