Diagnose a pipeline error using the perfect input test on a misclassified dev set image.
Case context: An engineer's cat detector -> cat breed classifier pipeline misclassifies an image of a Siamese cat as 'not a cat' (y=0). The engineer suspects either the detector's bounding box or the breed classifier is at fault, but the pipeline only reports the final wrong answer, not which component erred.
Question: Describe how the engineer should use the perfect input test to determine which component to fix, and explain how each possible test outcome should be interpreted.
Sample answer: The engineer should manually hand-label a correct bounding box around the cat in the image, then crop the image using that hand-labeled box and feed it into the cat breed classifier alone, bypassing the detector's actual output. If the classifier still outputs y=0 despite being given the perfect crop, the classifier itself is misclassifying the Siamese cat, so effort should go toward improving the classifier. If instead the classifier correctly outputs y=1 with the perfect crop, then the classifier works fine and the original error was caused by the detector producing a poor bounding box, so effort should go toward improving the detector.
Key points:
- Hand-label a perfect bounding box for the specific failing image
- Crop and feed that perfect input into the classifier alone
- Classifier still wrong -> classifier is at fault
- Classifier now correct -> detector's original output was at fault
Rubric: Full credit: correctly describes replacing the detector output with a hand-labeled bounding box, running it through the classifier, and correctly attributes fault for both possible outcomes. Partial credit: describes the substitution but misattributes one outcome. No credit: proposes an unrelated diagnostic approach.
0
1
References
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Machine Learning Yearning (Deeplearning.ai)
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Attributing Dev Set Errors to Pipeline Components
In the perfect input test for a cat detector -> cat breed classifier pipeline, what is used to replace the upstream component's output?
True or False: If the cat breed classifier still misclassifies an image given a hand-labeled perfect bounding box, the cat detector is blamed for the error.
In the perfect input test, if the breed classifier now correctly outputs y=1 given a hand-labeled bounding box, attribute the error to the _____.
Match each perfect input test outcome to the component that should be blamed for the pipeline error.
Order the steps of the perfect input test used to attribute a pipeline error between a cat detector and a cat breed classifier.
Explain why substituting a hand-labeled perfect output isolates which pipeline component is responsible for an error.
Diagnose a pipeline error using the perfect input test on a misclassified dev set image.
What must an engineer create to conduct the perfect input test, and why must it be 'perfect'?
Which conclusion follows if the cat breed classifier misclassifies an image even when given a hand-labeled perfect bounding box?
True or False: The perfect input test can be used to attribute an error to exactly one component in a multi-step pipeline.