Question: Using the concept of Optimization Verification, explain why a team observing 95% of errors due to the scoring function and 5% due to the optimization algorithm should prioritize improving score estimation over optimization. Discuss what upper bound the 5% figure places on optimization improvements.

Sample answer: Optimization Verification attributes dev set errors to either the scoring function or the optimization algorithm. When only 5% of errors are due to the optimization algorithm, even a perfect optimization procedure could realistically eliminate only about 5% of total errors, since that is the maximum share attributable to that component. In contrast, the scoring function accounts for 95% of errors, so improving how the score is estimated has a much higher ceiling for reducing overall error. Prioritizing effort based on this diagnostic ensures engineering time is spent where it can have the greatest impact.

Key points:

• Optimization Verification attributes errors between the scoring function and optimization algorithm
• 5% error share caps the realistic improvement possible from optimizing the algorithm
• 95% error share on scoring function indicates the highest-impact area for improvement
• Effort should be redirected to how the score is estimated
• The diagnostic result should directly inform prioritization decisions

Rubric: Full credit requires explaining that the error share sets an upper bound on potential improvement, correctly identifying the scoring function as the priority, and referencing the 95%/5% figures.