Google

When starting a new project, especially in an area in which one is not an expert, it is hard to correctly guess the most promising directions. Rather than trying to design and build the perfect system at the outset, one should build and train a basic system as quickly as possible, perhaps in a few days, and then use error analysis to identify the most promising directions and iteratively improve the algorithm.

Build a Basic System Quickly and Iterate Using Error Analysis

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A new email anti-spam system can present many possible development directions, making it difficult to choose the best initial direction.

Email Anti-Spam System as a New System Example

The advice to build a basic system quickly is meant for readers who want to build AI applications, rather than readers whose goal is to publish academic papers.

Quick Basic-System Advice Targets AI Applications

What is the recommended first step when starting a new ML project in an unfamiliar area?

True or False: You should spend significant time designing the perfect ML system before building or training anything.

After building a basic ML system quickly, you should use _____ to identify the most promising directions for iterative improvement.

Why does Machine Learning Yearning recommend building a basic system quickly rather than designing a perfect system at the outset?

According to Machine Learning Yearning, error analysis should be performed before building any initial system in order to avoid wasted effort.

Machine Learning Yearning recommends building and training a basic system as quickly as possible—perhaps in just a _____ days.

Match each phase of the quick-iteration workflow to its purpose as described in Machine Learning Yearning.

Order the steps of Machine Learning Yearning's recommended workflow for starting a new ML project from beginning through iteration.

What key value does Machine Learning Yearning say you gain by examining a basic system, even when it is far from the best system you could build?

Machine Learning Yearning states that experienced ML practitioners can reliably identify the best project direction before building any system.

After building a basic system, Machine Learning Yearning recommends using _____ to identify the most promising directions and iteratively improve the algorithm.

Match each Machine Learning Yearning recommendation to the reasoning the book provides for it.

Order the reasoning steps that justify the quick-build-and-iterate strategy recommended in Machine Learning Yearning.

Question:
Explain why trying to design and build a perfect machine learning system at the start of a project in a new domain is typically counterproductive. Use the concepts of domain expertise, predicting project direction, and error analysis to support your analysis.

Sample answer:
Designing a perfect system from the start is difficult because when beginning a project, especially in a new domain where one is not an expert, it is hard to correctly guess the most promising directions. Instead of trying to build the perfect system, it is more effective to quickly build and train a basic system (potentially in a few days). Examining this basic system, even if it is suboptimal, provides crucial clues. By running error analysis on its output, you can identify the most promising directions and iteratively improve the algorithm.

Key points:
- Difficulty in guessing the most promising directions in a new domain
- Avoiding the trap of trying to design/build the perfect system at the start
- Building and training a basic system quickly (within a few days)
- Using error analysis to guide iterative improvement

Rubric:
The response should address: 1) the difficulty of guessing promising directions in a new domain, 2) the risk of wasting time building a complex but incorrect system at the outset, and 3) how error analysis on a basic system provides data-driven directions for iteration.

Analyzing the Pitfalls of Building a Perfect System at the Outset

Case context:
Your team is starting a new email anti-spam project. The team has several competing ideas: building a custom spelling checker, implementing a sender reputation system, or using complex word-frequency analysis. The team does not have deep domain expertise in spam filtering and is arguing over which direction to design first.

Question:
Based on the principles of rapid prototyping, what should the team do first to resolve this disagreement and identify the most promising direction?

Sample answer:
The team should not spend time trying to design the perfect system at the outset. Instead, they should build and train a basic anti-spam system as quickly as possible, perhaps in just a few days. Even though this basic system will be far from the best system, they can examine how it functions and perform error analysis on its mistakes. This will quickly provide clues showing which direction is actually the most promising to invest their time in.

Key points:
- Avoid trying to design the perfect system or argue over directions without data
- Build and train a basic anti-spam system quickly (in a few days)
- Use error analysis on the basic system to find clues identifying the most promising direction

Rubric:
The answer should recommend building a basic, suboptimal system quickly (in a few days) instead of designing the perfect system first, and specify that error analysis on this basic system will provide the clues needed to choose the most promising direction.

Resolving Strategy Disagreements in a New Anti-Spam Project

Question:
According to Machine Learning Yearning, why is it valuable to examine how a basic, suboptimal system functions, even when it is far from the best system you can build?

Sample answer:
Examining a basic, suboptimal system is valuable because it provides clues that show the most promising directions in which to invest your time.

Key points:
- Provides functional clues by examining the basic system's behavior
- Identifies the most promising directions to invest time

Rubric:
The answer must state that examining the system provides clues or information to identify the most promising directions for investing time or iterative improvement.

Learn Before

Related