Question: A photo app is evaluated using a dev/test set made mostly of full-grown dogs, but after launch many users upload puppy photos. Explain why the original dev/test set may be misleading.

Sample answer: The dev/test set was built around one kind of input: images of adult dogs. That makes it a reasonable check for performance on that population, but it does not match what the app actually receives after deployment if puppy photos become common. Because the real input distribution is different, performance on the original dev/test set may not predict real-world behavior. In that situation, the dev/test set is not representative of the target distribution.

Key points:

• The dev/test data mostly contains adult dog images
• Real users upload many puppy images
• The test distribution differs from actual use
• A mismatch makes the dev/test set unrepresentative

Rubric: The response must explain that a dev/test set built around adult dogs does not reflect real usage if the deployed app receives many puppy images, and it must clearly conclude that the dev/test distribution is not representative of the actual data distribution.