A large holdout split is useful when data is limited, and an older rule of thumb was to reserve about 30% of the data for testing. That kind of split can be reasonable for small to medium datasets, such as those with roughly 100 to 10,000 examples. When datasets reach much larger scales, it is usually better to allocate a much smaller percentage to dev and test sets. The key requirement is that these sets be large enough to give a dependable estimate of performance, but no larger than needed for that purpose.

Dev and Test Splits Should Be Smaller for Very Large Datasets

A test set should contain enough examples to give a reliable estimate of the system's final performance.

A Test Set Large Enough for a Reliable Final Check

The test set should contain enough examples to make the estimate of overall system performance reliable.

Test Set Size Should Enable Reliable Evaluation

Which use of a test set matches its intended role?

A final test set can measure performance without being used to choose model settings.

The held-out test set is used to _____ the final model’s performance on unseen data.

Match each holdout-set concept with its proper role.

Order the steps for deciding how a test set may be used.

Question:
In a short analytical response, describe the proper role of a test set and name the kinds of decisions it should not drive.

Sample answer:
A test set is meant to estimate how well the final model will perform on unseen data. It should not be used to choose between learning algorithms or to tune hyperparameters. Those decisions should be made using training or validation data, while the test set is reserved for final evaluation.

Key points:
- The test set is for estimating generalization performance.
- It should not determine which learning algorithm is chosen.
- It should not determine hyperparameter settings.
- Model selection and final evaluation serve different purposes.

Rubric:
A strong response clearly identifies the test set as an evaluation tool, excludes both algorithm choice and hyperparameter tuning, and separates selection from final assessment.

What a test set is for, and what it is not for

Case context:
A data science team trains several models, compares their results on a held-out test set, and then uses those same test results to choose the final model and tune its hyperparameters.

Question:
What is wrong with this workflow, and what should the test set be used for instead?

Sample answer:
The team is using the test set for decisions it should not influence. Model selection and hyperparameter tuning should be based on training or validation data, while the test set should be saved for a final, unbiased evaluation after all choices are finished.

Key points:
- Do not use the test set to choose among models.
- Do not use the test set to tune parameters.
- Use the test set only for final performance evaluation.

Rubric:
The response must identify both improper uses of the test set—model selection and parameter tuning—and explain that the test set should be reserved for final evaluation.

Using the Right Evaluation Set for Model Selection

Question:
Answer in one to three sentences.

Sample answer:
A test set is kept aside so it can give a final estimate of how well a trained model will generalize to new data. It should not be used to choose among candidate models or to adjust hyperparameters.

Key points:
- Provides a final estimate of generalization.
- Do not use it to select the model.
- Do not use it to tune hyperparameters.

Rubric:
The answer should describe the final evaluation purpose and reject both model selection and hyperparameter tuning.

Test Set Purpose and Limits

Using the final holdout set appropriately

The test set should be used to choose the model you deploy.

A test set is used to estimate how well a trained system performs, but it should not be used to choose the model, features, or hyperparameters.

Google

Machine learning is a branch of artificial intelligence focused on building systems that improve their task performance by learning from data or other experience. It draws on ideas from statistics, optimization, and data analysis to make predictions, recognize patterns, and support decisions.

Machine Learning

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Supervised learning is a method for learning a mapping from inputs x to outputs y from labeled examples (x, y). Common supervised learning algorithms include linear regression, logistic regression, and neural networks.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

A development set, sometimes called a holdout validation set, is a data split used to tune hyperparameters, compare feature choices, and make other model-design decisions during development.

Development Set

Purpose of a Test Set

When a project must satisfy more than one evaluation criterion, a common strategy is to set one metric as a minimum acceptable requirement and then optimize a different metric among the solutions that meet that requirement. For example, a team might require inference time to stay under a limit and then select the model with the highest accuracy.

Combining Metrics With a Threshold and an Objective

For a brand-new machine learning task, it is usually very hard to know ahead of time which method will work best. Even skilled researchers often test many different ideas before they find one that performs well enough.

It Is Hard to Predict the Best Machine Learning Strategy Before Trying It

Machine learning development usually proceeds in cycles. A practical loop is to propose an approach, implement it, run an experiment to measure the result, and then use what you learned to choose the next change. Because each round reveals new information, teams often test many ideas before settling on one that works well.

An Iterative Machine Learning Workflow

In machine learning, two broad causes of poor performance are bias and variance. Separating them helps you choose the right next step, such as changing the model, gathering more data, or adjusting the training approach.

Bias and Variance as Two Major Sources of Error

With enough labeled input-output examples, an end-to-end model can learn to map inputs directly to outputs that are more complex than a single number. Those outputs can include text, images, audio, or other structured results.

What End-to-End Learning Can Predict

A pipeline can be poorly designed if the full system performs much worse than human level even though each stage is close to human level on the information it receives. In that case, the weakness is often that important information is being lost, so the design should be reconsidered.

When a Pipeline Is Missing Information

Text polarity detection is the task of deciding whether a piece of text expresses a favorable or unfavorable opinion, such as determining whether a product review praises or criticizes the item.

Text Polarity Detection

Machine learning strategy is the discipline of deciding which changes are most likely to improve a model, based on evidence from errors, data splits, and project constraints. It helps you prioritize the next experiment instead of guessing blindly.

Machine Learning Strategy

Recent progress in deep learning has mainly come from larger data sets and greater computing power.

Learn Before

Related

Learn After