Use component error counts to set improvement priorities.
Question: A machine-learning team inspects 100 incorrect validation examples from a document-classification pipeline. They trace 84 errors to the document parser and 16 errors to the topic classifier. Explain how these counts should guide which part of the system receives the next round of engineering effort.
Sample answer: The team should use the error counts to identify the main source of failure. Because the document parser accounts for 84 of the 100 mistakes while the topic classifier accounts for only 16, the parser is the larger bottleneck. The team should therefore prioritize improving the parser, since fixing the component that causes most of the errors is likely to produce the biggest gain.
Key points:
- Assign each validation mistake to the component that produced it.
- Compare the counts across components.
- Prioritize the component responsible for most of the errors.
Rubric: The response must state that error counts show which pipeline component is failing most often, identify the document parser as causing 84 errors and the topic classifier as causing 16, and conclude that improving the parser should be the priority.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Example Attribution Can Support a Second Pass of Debugging
Which component should receive the most attention if 72 of 80 dev-set mistakes come from the signal preprocessor?
A single source of most validation errors should receive priority.
Identify the component with the most assigned errors
Interpreting Component Error Counts
Order the steps for using component error counts to prioritize pipeline fixes.
A speech-recognition pipeline shows 9 times as many dev-set mistakes in the wake-word detector as in the speaker-id module. What should the team do next?
If one subsystem is responsible for 10 out of 100 validation mistakes, it should be the first area the team improves.
Reviewing 100 incorrect validation predictions and assigning each one to a pipeline _____ shows which step should be improved first.
Match each pipeline error-analysis term to its correct description.
Order the steps used to decide which pipeline component deserves improvement first.
Use component error counts to set improvement priorities.
Using component error counts to decide where to improve a document-processing pipeline.
Choose the component to improve when most dev-set mistakes come from one stage.