Learn Before
Sizing a dev set for a high-revenue product recommendation engine.
Case context: You are leading the machine learning team for a major e-commerce platform's product recommendation engine. The current model is already highly optimized. An engineer proposes testing a new algorithm that they estimate will improve accuracy by just 0.01%, but they note that a dev set of 10,000 examples won't be able to confirm this.
Question: Based on Ng's principles, should you invest in evaluating this tiny improvement, and what must you do to the dev set to test it properly?
Sample answer: Yes, the improvement should be evaluated because for mature, high-value applications like product recommendations, a 0.01% gain directly and significantly increases company profits. To test it, you must increase the dev set size to much larger than 10,000 examples to reliably detect such a small change.
Key points:
- 0.01% improvements are highly valuable in mature applications.
- Small improvements directly affect company profits.
- A dev set larger than 10,000 examples is needed to detect small improvements.
Rubric: The student must recognize that evaluating the 0.01% improvement is worthwhile due to its financial impact in mature applications, and conclude that the dev set size must be increased well beyond 10,000 examples.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why might a team building a high-value application need a dev set much larger than 10,000 examples?
Teams working on mature, high-value applications like advertising may actively pursue accuracy gains as small as 0.01%.
For high-value applications, the dev set may need to be much larger than _____ examples to detect very small improvements.
Match each high-value application domain to the reason why tiny accuracy improvements matter there.
Order the steps for deciding whether a standard 10,000-example dev set is sufficient for a high-value application.
According to Ng, why does a 0.01% accuracy improvement matter in domains like advertising or web search?
A dev set of exactly 10,000 examples is always large enough to detect accuracy improvements relevant to high-value applications like web search.
In high-value applications, teams may be motivated to find even a _____ improvement because it directly affects the company's profits.
Match each concept to its correct role in the reasoning for larger dev sets in high-value applications.
Arrange the reasoning steps Ng uses to justify larger dev sets in high-value applications, in the order he presents them.
Evaluate the need for ultra-large dev sets in mature business applications.
Sizing a dev set for a high-revenue product recommendation engine.
Financial motivation for scaling dev sets beyond 10,000 examples.