Case Study

Diagnose a pipeline error using the perfect input test on a misclassified dev set image.

Case context: An engineer's cat detector -> cat breed classifier pipeline misclassifies an image of a Siamese cat as 'not a cat' (y=0). The engineer suspects either the detector's bounding box or the breed classifier is at fault, but the pipeline only reports the final wrong answer, not which component erred.

Question: Describe how the engineer should use the perfect input test to determine which component to fix, and explain how each possible test outcome should be interpreted.

Sample answer: The engineer should manually hand-label a correct bounding box around the cat in the image, then crop the image using that hand-labeled box and feed it into the cat breed classifier alone, bypassing the detector's actual output. If the classifier still outputs y=0 despite being given the perfect crop, the classifier itself is misclassifying the Siamese cat, so effort should go toward improving the classifier. If instead the classifier correctly outputs y=1 with the perfect crop, then the classifier works fine and the original error was caused by the detector producing a poor bounding box, so effort should go toward improving the detector.

Key points:

  • Hand-label a perfect bounding box for the specific failing image
  • Crop and feed that perfect input into the classifier alone
  • Classifier still wrong -> classifier is at fault
  • Classifier now correct -> detector's original output was at fault

Rubric: Full credit: correctly describes replacing the detector output with a hand-labeled bounding box, running it through the classifier, and correctly attributes fault for both possible outcomes. Partial credit: describes the substitution but misattributes one outcome. No credit: proposes an unrelated diagnostic approach.

0

1

Updated 2026-07-09

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI