Why a Mismatched Validation Set Can Waste Effort

Which explanation is NOT one of the standard reasons a model can score well on the dev set but poorly on the test set when the two sets come from different distributions?

True or False: If a development set and a test set come from different distributions, it is easy to pinpoint the reason a model scores worse on the test set.

If the development set and the test set come from different _____, a performance gap is hard to interpret.

Match each distribution-shift situation to the most likely explanation.

Order the three explanations for why a model can look strong on development data but weak on test data when the data distributions are different.

When the test data are drawn from a tougher distribution than the development data, what is the most reasonable conclusion?

True or False: If a model scores lower on the test set than on the dev set, then the test set must be inherently more difficult.

A model may do well on the _____ set and still perform poorly on the test set if the two sets come from different distributions.

Match each dev/test diagnosis to the most appropriate next step.

Arrange the steps a practitioner follows when a spam filter succeeds on validation data but fails after deployment.

Why a good development score can still leave test results unexplained

Why a Model Looks Strong in Validation but Weak on Deployment Data

Three Reasons a Held-Out Test Set Can Look Worse