Question: A ranking system assigns scores to two candidate outputs for the same input: one correct and one incorrect. Why must the correct output receive a strictly larger score than the incorrect one in the diagnostic check, rather than merely tying it? What does a tie tell you?

Sample answer: The check is meant to verify that the scoring component can actually prefer the correct output. A tie does not show that preference, so it does not pass the test. The correct output must score strictly higher to demonstrate that the score separates better outputs from worse ones. If the scores are equal, the diagnostic fails because the scoring function has not distinguished the two candidates. In that case, the right next step is to improve how the score is learned, estimated, or calibrated.

Key points:

• The correct output must score strictly higher, not just equal
• A tie means the score does not separate correct from incorrect outputs
• Equal scores should be treated as a diagnostic failure
• The appropriate response is to improve the scoring component

Rubric: Full credit explains that equality is not enough and correctly interprets a tie as a failure of the scoring function; partial credit states that strict inequality is required without explaining the meaning of a tie; no credit for responses that ignore the distinction between equal and strictly larger scores.