Case context: A product team keeps two evaluation subsets: a Review set for manual inspection and a Locked set for automated scoring only. After a release candidate scores poorly on the Locked set, an engineer opens several Locked-set cases and studies them one by one to understand the failures.

Question: What methodological mistake did the engineer make, why is it a problem for the Locked set, and what should the engineer have inspected instead?

Sample answer: The mistake was manually examining examples from the Locked set. That set is meant to stay untouched except for automated measurements, so looking at individual cases breaks its role as a reserved evaluation set. The engineer should have done the manual error review on the Review set instead, while keeping the Locked set for unbiased scoring and comparison.

Key points:

• Manual inspection of the Locked set is the error.
• The Locked set should remain reserved for automated evaluation only.
• Human debugging and error analysis belong in the Review set.

Rubric: The answer must identify that opening the Locked set is the mistake. It must explain that this violates the set's purpose as an automated-only evaluation subset. It must recommend using the Review set for manual inspection instead.