Case Study

Diagnose why a system trained on website images underperforms in the field.

Case context: A team building an image classifier had no way to collect data resembling the images their product would actually encounter in the field. Following common practice, they started by training on a large set of website images. After deployment, the system performs noticeably worse on real field images than it did on their test set.

Question: Based on the source material, what is the most likely explanation for this performance gap, and what does it suggest about the original decision to use website images?

Sample answer: The most likely explanation is the generalization risk explicitly warned about in the source: website images can differ from real future data, so a system trained on them may not generalize well to actual field conditions. This does not necessarily mean the original decision was wrong, since the team had no way to get approximating data, but it confirms the anticipated risk materialized and highlights the need to seek more representative data going forward.

Key points:

  • Website images may differ from real future data
  • The source predicts exactly this generalization risk
  • The original decision may still have been reasonable given no alternatives existed
  • The gap signals a need to pursue more representative data

Rubric: Full credit: response connects the performance gap to the generalization risk from the source and acknowledges the decision was reasonable given the lack of alternatives. Partial credit: response identifies the risk but doesn't connect it to the case details. No credit: response invents unrelated causes not grounded in the source.

0

1

Updated 2026-07-11

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI