Learn Before
Judge whether a team should code a new model immediately or first inspect a sample of mistakes.
Case context: A machine learning team has an idea for a new image model. They are excited and want to spend the next four weeks building it right away. Another engineer suggests first reviewing 100 recent misclassifications by hand, which would take about two hours, to see whether the idea addresses the real failure modes.
Question: What should the team do, and what is the main danger of ignoring the engineer's suggestion?
Sample answer: The team should pause and inspect the 100 misclassified images before starting the month-long build. The review takes very little time compared with four weeks of development, and it can prevent the team from investing heavily in a change that does not fix the actual problems. If they skip the review, they may spend a month building something that ends up improving the model very little or not at all.
Key points:
- Do the short manual error review first.
- Do not jump straight into a long implementation effort.
- The risk is spending a month on a change with little or no payoff.
Rubric: The response must recommend the manual review and explain the risk of losing a month of effort on a change that does not meaningfully help. It should also contrast the short review time with the much larger implementation cost.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Why do many engineers hesitate to do a quick error review?
Checking 120 mistaken predictions at 45 seconds each takes less than two hours.
A small investment in analysis can prevent large waste.
Match the Diagnostic Strategy Terms
Order the events that show why skipping error analysis can waste effort.
What is a likely downside of implementing a model change before reviewing the main errors?
Most ML engineers prefer spending time on error analysis over trying new implementation ideas.
How Many Examples Can Be Reviewed Quickly?
Match each time estimate to what it represents in the manual review argument.
Order the argument for checking errors before committing to a design choice.
Why Teams Put Off Error Review
Judge whether a team should code a new model immediately or first inspect a sample of mistakes.
Why Do Engineers Skip a Quick Error Review?