1Cademy - Diagnosing Dev Set Errors Using Optimization Verification Inequalities

Learn Before

Applying Optimization Verification Across Dev Set Errors

Case Study

Diagnosing Dev Set Errors Using Optimization Verification Inequalities

Case context: While analyzing a specific dev set error, you evaluate the scoring function $Score_A(.)$ for both the correct output $S^*$ and the algorithm's output $S_{out}$ . The results are $Score_A(S^*) = -15$ and $Score_A(S_{out}) = -12$ .

Question: Based on these scores, how should you classify this error using the Optimization Verification test, and what does this tell you about your system?

Sample answer: The error should be classified as a scoring-function error because $Score_A(S^*) \le Score_A(S_{out})$ ( $-15 \le -12$ ). This indicates that the optimization algorithm successfully maximized the scoring function, but the scoring function itself is faulty because it gave a higher score to the incorrect output than the correct output.

Key points:

Classify the error as a scoring-function error.
Identify that the condition $Score_A(S^*) \le Score_A(S_{out})$ holds since $-15 \le -12$ .
Conclude that the scoring function is at fault, not the optimization algorithm.

Rubric: The response must identify the error as a scoring-function error, cite the relevant inequality condition ( $Score_A(S^*) \le Score_A(S_{out})$ ), and explain that the scoring function is at fault rather than the optimization algorithm.

0

1

Updated 2026-05-26

Contributors are:

Who are from:

References

Learn Before

Related