Learn Before
Diagnose a team that uses test results to choose an algorithm and its parameters.
Case context: A team evaluates several learning algorithms on a test set, then uses those results to choose both the algorithm and its parameters.
Question: What is inappropriate about the team's use of the test set, and what role should the test set serve instead?
Sample answer: The team is improperly using the test set to make decisions about the learning algorithm and its parameters. The test set should instead be reserved for evaluating algorithm performance without guiding those choices.
Key points:
- Algorithm choice should not be based on the test set.
- Parameter choice should not be based on the test set.
- The test set's proper role is evaluating performance.
Rubric: The response must identify both improper decisions—algorithm selection and parameter selection—and state that the proper role is performance evaluation.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Machine Learning
Deep Learning
Supervised Learning
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
70/30 Train/Test Split Heuristic Does Not Apply to Large Datasets
Sizing the Test Set for a Confident Performance Estimate
Test Set Should Support High-Confidence Performance Estimates
Which use of a test set is consistent with its stated purpose?
A test set may evaluate an algorithm without helping select its parameters.
The test set is used to _____ algorithm performance.
Match each test-set concept with its correct role.
Order the reasoning for deciding whether a proposed test-set use is appropriate.
Explain the boundary between evaluating with a test set and making decisions from it.
Diagnose a team that uses test results to choose an algorithm and its parameters.
What is the test set's role, and what two decisions must it not guide?
Which proposal misuses the test set?
Test-set performance should determine which learning algorithm is used.