Case context: A learning algorithm's dev set errors were analyzed using Optimization Verification. The results showed that 95% of the errors were due to the scoring function, Score_A(.), and only 5% were due to the optimization algorithm. The team has limited engineering time and must decide whether to invest primarily in improving the optimization procedure or in improving how the score is estimated.

Question: Based on the Optimization Verification results, what should the team decide, and why? What is the realistic ceiling on error reduction if they instead invest heavily in the optimization algorithm?

Sample answer: The team should invest primarily in improving how the score is estimated, since 95% of errors are attributable to the scoring function. If they instead invest heavily in improving the optimization algorithm, they could realistically eliminate only about 5% of the total errors, because that is the maximum share of errors attributable to that component. Investing in score estimation gives a much higher potential payoff.

Key points:

• Team should prioritize improving score estimation
• 95% of errors are attributable to the scoring function
• Optimization algorithm improvements are capped at about 5% error reduction
• Decision should be grounded directly in Optimization Verification results

Rubric: Full credit requires correctly recommending improvement of score estimation, citing the 95% figure, and correctly stating the 5% ceiling on optimization-based improvements.