Case context: A team trains a document classifier using scanned files collected from a desktop portal and evaluates it on a held-out set from the same portal. The model performs well in that setting. After deployment in a mobile app, where users submit handheld photos of documents, accuracy drops sharply. The phone images are often tilted, unevenly lit, and lower in resolution than the scanned files.

Question: What should the team identify as the main issue, and why did the earlier evaluation miss it?

Sample answer: The main issue is a distribution mismatch between the scanned documents used for training and testing and the handheld phone photos seen after deployment. The earlier evaluation missed the problem because the test set came from the same scan-based source as the training data, so it did not reflect the image conditions the app would actually receive.

Key points:

• Scanned documents were used for development data.
• Handheld phone photos were used in deployment.
• The two sources differed in resolution, lighting, and angle.
• The test set did not match the real deployment distribution.

Rubric: Full credit requires identifying the distribution mismatch, distinguishing scans from phone photos, and explaining why a scan-based test set could not reliably predict performance on the deployed inputs.