Google

Once a dev set and test set are defined, a team can try many ideas, such as different learning algorithm parameters, to see what works best. The dev and test sets allow the team to quickly see how well the algorithm is doing, and their purpose is to direct the team toward the most important changes to make to the machine learning system. Measuring each idea's performance on the dev set lets the team quickly decide whether they are heading in the right direction, and helps them identify which ideas to keep refining and which ones to discard.

Purpose of Dev and Test Sets

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

During development, a team will try many ideas about algorithm architecture, model parameters, and choice of features. Having a single-number evaluation metric such as accuracy allows the team to sort models according to their performance on the metric and quickly decide what is working best. It also speeds up decisions when selecting among a large number of classifiers by giving a clear preference ranking and a clear direction for progress.

Single-Number Evaluation Metric

Having an initial dev/test set and metric helps a team iterate quickly.

Initial Dev/Test Set and Metric Helps Quick Iteration

If the initial dev/test set or metric missed the mark, change it quickly. One warning sign is that the dev set plus metric ranks one classifier higher while the team thinks another classifier is superior for the product. Possible causes include a dev/test set distribution that differs from the actual distribution of interest, overfitting to the dev set, or a metric that measures something other than what the project needs to optimize. After changing dev/test sets or metrics, make sure the team knows the new direction.

Changing Dev/Test Sets or the Metric When They No Longer Guide the Team

Error analysis uses data about an ML system?s mistakes to derive insight about what to do next. There is no single right way to carry it out; common design patterns help, but teams can experiment with other ways of analyzing errors.

Error Analysis

When deciding to improve label quality, consider double-checking the labels of both examples that the system misclassified and examples it correctly classified. It is possible that both the original label and the learning algorithm were wrong on an example.

Double-Checking Labels of Both Misclassified and Correctly Classified Dev Examples

What is the primary purpose of the dev and test sets once they are defined?

True or False: Measuring an idea's performance on the dev set lets a team quickly decide if it is heading in the right direction.

The dev and test sets allow a team to quickly see how well their _____ is doing.

Match each action related to dev/test sets with its described effect.

Order the steps of using dev/test sets to guide iteration on a machine learning idea.

Question:
In a few sentences, explain why having a dev set, test set, and metric in place is important for a machine learning team, based on how it affects their ability to evaluate ideas.

Sample answer:
Having dev/test sets and a metric is important because it lets a team measure the performance of each idea they try, such as new algorithm parameters, on the dev set. This quick measurement lets the team see whether they are heading in the right direction and quickly detect which ideas produce small or large improvements. As a result, the team can efficiently decide which ideas to keep refining and which to discard, directing their effort toward the most important changes to the system.

Key points:
- Dev/test sets and a metric allow measuring performance of each idea tried
- This lets the team quickly see if they're heading in the right direction
- It helps detect small or large improvements from an idea
- It helps the team decide what to keep refining and what to discard

Rubric:
Full credit: explains that dev/test sets and a metric allow quick measurement of idea performance, enable fast decisions about direction, and help the team decide what to refine versus discard. Partial credit: mentions only one of these effects. No credit: does not connect dev/test sets to evaluating ideas.

Explain why having dev/test sets and a metric is important for a team's workflow.

Case context:
A machine learning team is experimenting with several different learning algorithm parameters, but they have not defined a dev set or test set for their project. After weeks of tweaking, the engineers disagree about which changes actually helped.

Question:
Based on the purpose of dev and test sets described in the source, diagnose what the team is missing and explain how defining a dev set and test set would resolve their disagreement.

Sample answer:
The team is missing a dev set and test set that would let them quickly and objectively measure how well the algorithm is doing after each change. Without this, they have no reliable way to see if they're heading in the right direction, so engineers are left guessing or disagreeing about results. By defining a dev set and test set, the team could measure each idea's performance on the dev set, quickly detect which ideas give small or large improvements, and confidently decide which ideas to keep refining and which to discard.

Key points:
- Without a dev/test set, the team cannot quickly see how well the algorithm is doing
- This lack of measurement is why engineers disagree about which changes helped
- Defining dev/test sets allows measuring each idea's performance on the dev set
- This directs the team toward the most important changes and shows what to keep or discard

Rubric:
Full credit: identifies the missing dev/test set as the cause of the team's inability to evaluate ideas quickly, and explains that defining these sets would allow quick performance measurement and clear decisions about which ideas to pursue. Partial credit: identifies the missing dev set but does not explain the resulting benefit. No credit: does not connect the scenario to the purpose of dev/test sets.

A team tries several algorithm changes without a dev set. Diagnose the resulting problem.

Question:
In one to three sentences, describe how measuring performance on the dev set helps a team decide which ideas to keep refining and which to discard.

Sample answer:
Measuring an idea's performance on the dev set lets the team quickly detect whether that idea gives a small or large improvement. This quick feedback allows the team to decide efficiently which ideas are worth continuing to refine and which ones should be discarded.

Key points:
- Dev set measurement quickly detects small or large improvements from an idea
- This lets the team decide which ideas to keep refining and which to discard

Rubric:
Full credit: mentions that dev set measurement quickly reveals improvement (or lack thereof) and that this informs the decision to refine or discard an idea. Partial credit: mentions only one of these points. No credit: unrelated or missing key concept.

Learn Before

Related