Question: When you review every mistake made on a validation set, how can you decide whether each case points to a model-training problem or to a scoring/metric problem? Explain the check using the ideal output $S^{\star}$ and the system output $S_{out}$.

Sample answer: Go through all validation errors one by one. For each case, compute the score for the ideal output, $Score_A(S^{\star})$, and the score for the model’s output, $Score_A(S_{out})$. If $Score_A(S^{\star}) > Score_A(S_{out})$, then the model should have been able to do better under this scoring rule, so classify that error as a model-training or optimization issue. If $Score_A(S^{\star}) \le Score_A(S_{out})$, then the scoring rule is not separating the ideal output from the model output in that case, so classify it as a scoring-function issue.

Key points:

• Review the full set of validation errors, not just a few examples.
• Compare the score of the ideal output with the score of the model output for each error.
• Label cases where the ideal output scores higher as model-training issues.
• Label cases where the ideal output does not score higher as scoring-function issues.

Rubric: The answer should explain that every validation error is checked, $Score_A(S^{\star})$ is compared with $Score_A(S_{out})$, and the result is used to separate model-training issues (when $Score_A(S^{\star}) > Score_A(S_{out})$) from scoring-function issues (when $Score_A(S^{\star}) \le Score_A(S_{out})$).