Case context: A team reports that its algorithm achieves 85% accuracy on training examples it has seen and 95% accuracy on dev/test examples it has not seen.

Question: Based only on the source principle, what should the team diagnose about this report, and what performance must be addressed first?

Sample answer: The team should diagnose the reported relationship as inconsistent with the source principle. Dev/test performance is usually worse than training-set performance, so an algorithm achieving only 85% on seen examples cannot be achieving 95% on unseen examples. The team should first address and improve training-set performance before expecting strong dev/test performance.

Key points:

• The reported performance relationship is inconsistent with the source.
• The algorithm scores lower on seen examples than it allegedly does on unseen examples.
• Dev/test performance is usually worse than training performance.
• Training-set performance should be addressed first.

Rubric: Full credit requires identifying the report as inconsistent, connecting training data with seen examples and dev/test data with unseen examples, and stating that training-set performance must be addressed first.