Question: In a concise essay, explain why a random 70%/30% split into training and test sets can fail when the training distribution differs from the distribution the system ultimately needs to perform well on.

Sample answer: A random split only preserves the characteristics of whatever data was collected. If that data (e.g., website images) does not represent the target use case (e.g., mobile phone images), then even a perfectly executed random split will produce a test set that reflects the wrong distribution. As a result, performance on the test set won't accurately predict performance on the real target distribution, undermining the split's usefulness for evaluation.

Key points:

• Random split preserves the distribution of available data
• Available data distribution may differ from target-use distribution
• Example: website images vs. mobile phone images
• Test set becomes unrepresentative of the real target use case
• This undermines the split's value for evaluating real-world performance

Rubric: Full credit requires explaining that a random split preserves the source data's distribution, noting the mismatch example (website vs. mobile images), and explaining the consequence for test set validity.