Learn Before
When to Attribute an Error to Optimization
Question: For a fixed input, when should a failure be blamed on the optimization procedure rather than on the model or scoring rule?
Sample answer: The optimization step should be blamed when the intended answer gets a higher score than the answer the system produced: Score_x(y^*) > Score_x(y_{out}).
Key points:
- Compare the score of the intended target answer with the score of the produced answer.
- If the target answer scores higher, the search procedure missed a better option.
- The condition can be written as Score_x(y^*) > Score_x(y_{out}).
Rubric: The answer must state that optimization is at fault when the target output has a higher score than the actual output.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Classifying development-set mistakes with optimization verification
Approximate Scoring Followed by Approximate Search
Checking Whether a Translation Error Comes from Search or Scoring
Checking Reward Quality in Reinforcement Learning
A useful human output for optimization verification
What does this optimization check suggest when the reference candidate scores higher than the algorithm’s output?
Using the Optimization Verification Test
Optimization Check Symbols
How to carry out an optimization verification check
Using the Optimization Verification Check
Debugging a Translation Decoder
When to Attribute an Error to Optimization
When Is an Optimization Check Useful?
Checking the Source of a Performance Problem
Optimization checks can isolate the source of a training problem