Learn Before
Evaluate the need for ultra-large dev sets in mature business applications.
Question: Analyze why applications like web search or advertising might require dev sets significantly larger than the standard 10,000 examples. In your analysis, discuss the relationship between small accuracy improvements and business outcomes.
Sample answer: For mature, high-value applications like advertising or web search, an accuracy improvement as small as 0.01% can have a massive direct impact on company profits. To reliably detect these tiny improvements rather than statistical noise, the dev set must be much larger than 10,000 examples.
Key points:
- High-value apps like advertising or search are highly profitable.
- A 0.01% improvement can directly impact company profits.
- A dev set much larger than 10,000 examples is required to measure such small gains.
Rubric: The response should identify that high-value applications profit significantly from tiny accuracy improvements (e.g., 0.01%) and explain that a larger dev set is necessary to statistically detect such minute changes.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Why might a team building a high-value application need a dev set much larger than 10,000 examples?
Teams working on mature, high-value applications like advertising may actively pursue accuracy gains as small as 0.01%.
For high-value applications, the dev set may need to be much larger than _____ examples to detect very small improvements.
Match each high-value application domain to the reason why tiny accuracy improvements matter there.
Order the steps for deciding whether a standard 10,000-example dev set is sufficient for a high-value application.
According to Ng, why does a 0.01% accuracy improvement matter in domains like advertising or web search?
A dev set of exactly 10,000 examples is always large enough to detect accuracy improvements relevant to high-value applications like web search.
In high-value applications, teams may be motivated to find even a _____ improvement because it directly affects the company's profits.
Match each concept to its correct role in the reasoning for larger dev sets in high-value applications.
Arrange the reasoning steps Ng uses to justify larger dev sets in high-value applications, in the order he presents them.
Evaluate the need for ultra-large dev sets in mature business applications.
Sizing a dev set for a high-revenue product recommendation engine.
Financial motivation for scaling dev sets beyond 10,000 examples.