Learn Before
Reviewing Both Missed and Correct Dev-Set Examples
When improving label quality, inspect not only the examples the model got wrong, but also some examples it got right. A dev example can contain a bad label even if the model's prediction matches that label, because the label and the learning algorithm may both be mistaken on the same case.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Single-Score Model Evaluation
An Early Dev/Test Setup Speeds Model Iteration
Revise the evaluation setup when it stops matching the goal
Error Analysis
Reviewing Both Missed and Correct Dev-Set Examples
After a development set and a test set have been chosen, what are they mainly used for?
True or False: A development set score can provide a quick check on whether a new idea is moving a model in the right direction.
Dev and test sets give a quick check on how well a team’s _____ is performing.
Match each dev/test-set action with its main effect.
Arrange the basic workflow for using development and test sets to improve a model.
Explain why a dev set, test set, and metric help a machine learning team work efficiently.
Diagnosing a project with no evaluation set
In one to three sentences, explain how a validation set can guide which model ideas deserve more work.
How does dev set performance help a team choose which ideas to keep improving?
What do teams usually try once dev and test sets are fixed?
What do teams typically do after they have set up dev and test sets?
True or False: With a fixed development set and a single evaluation metric, a team can quickly tell whether a new idea is helping by a little or by a lot.
A development set and a chosen evaluation metric help a team compare ideas quickly and see whether each change is making progress.
Learn After
Bias from Correcting Only the Mistakes
Which examples should be reviewed to improve dev-set label quality?
A single development example can have both an incorrect target label and an incorrect model prediction.
Review Both Error Cases and Correct Cases
Match each label-audit situation to the correct description.
Order the steps for a dev-set label audit.
Why can a dev example appear to be labeled correctly even when the label is wrong?
To assess label quality, it is enough to inspect only the examples your model got wrong.
Reviewing Labels on Development Examples
Match each label-review category to its role in checking data quality.
Why Correct Predictions Still Need Label Review
Why checking only mistakes can miss label problems
Why Reviewing Only the Flagged Errors Can Miss Label Problems
Why Recheck Apparently Correct Labels