Case context: A team is building a mobile app image classifier. Most of their labeled training images come from scraping websites, since website images are abundant and easy to collect. They randomly split this website-image dataset 70%/30% into training and test sets, following what they were taught was standard practice.

Question: What problem should this team diagnose with their train/test split approach, and why?

Sample answer: The team should recognize that their random 70/30 split is drawn entirely from website images, but the system ultimately needs to perform well on mobile phone images taken by users. Since the test set reflects the website-image distribution rather than the mobile-image distribution, strong performance on this test set will not reliably indicate strong performance on the actual target use case. The split fails to account for the distribution shift between training data and the real deployment environment.

Key points:

• Training and test data are both from website images
• Target use case is mobile phone images
• Random split does not correct for distribution mismatch
• Test performance won't predict real-world mobile performance

Rubric: Full credit identifies the distribution mismatch between website images (training/test source) and mobile phone images (target use), and explains that this makes the random split misleading for evaluation.