Once a dev set and test set are defined, a team can try many ideas, such as different learning algorithm parameters, to see what works best. The dev and test sets allow the team to quickly see how well the algorithm is doing, and their purpose is to direct the team toward the most important changes to make to the machine learning system. Measuring each idea's performance on the dev set lets the team quickly decide whether they are heading in the right direction, and helps them identify which ideas to keep refining and which ones to discard.

Purpose of Dev and Test Sets

Dev and test sets should be chosen to reflect data one expects to receive in the future and wants to perform well on. The test set should not simply be 30% of the available data when future data is expected to differ from the training data. One should not assume the training distribution is the same as the test distribution; examples should reflect what one ultimately wants to perform well on rather than only the data available for training.

Choosing Dev and Test Sets to Reflect Future Data

Once dev and test sets are defined, a team will focus on improving dev set performance, so the dev set should reflect the task one wants to improve on the most. Having different dev and test set distributions can lead to a system that works well on the dev set but does poorly on the test set. If the dev and test sets came from the same distribution, that result would have a clear diagnosis: the system has overfit the dev set, and the obvious cure is to get more dev set data. If the dev and test sets come from different distributions, the options are less clear: the system may have overfit the dev set, the test set may be harder than the dev set, or the algorithm might be doing as well as could be expected.

Choosing Dev and Test Sets from the Same Distribution When Possible

Changing dev/test sets or evaluation metrics during a project is common. Having an initial dev/test set and metric helps the team iterate quickly. If the dev/test sets or metric no longer point the team in the right direction, change them.

Changing Dev/Test Sets or Evaluation Metrics During a Project Is Common

For a new project, choosing dev/test sets quickly gives the team a well-defined target. A practical starting point is to create initial dev/test sets and an initial metric in less than one week, even if they are imperfect. This one-week timeline does not apply to mature applications; teams working on already-mature systems may spend months acquiring better dev/test sets.

Quickly Establishing Dev/Test Sets and Metric for New Applications

A dev set should be large enough to detect differences between the algorithms being tried. For example, a 100-example dev set would not be able to detect a 0.1 percentage-point accuracy difference such as 90.0% versus 90.1%. Dev sets of 1,000 to 10,000 examples are common, and a dev set could be much larger than 10,000 when teams want to detect even smaller improvements.

Sizing the Dev Set to Detect Meaningful Accuracy Changes

What is the primary purpose of a development (dev) set in a machine learning project?

The development set is sometimes called the 'hold-out cross validation set'.

The development set is sometimes also called the _____ cross validation set.

Match each development set role to its description in Machine Learning Yearning.

Order the steps for using a dev set to choose between two model configurations.

Which of the following is NOT a stated use of the development set in Machine Learning Yearning?

The dev set and the training set refer to the same data partition in Machine Learning Yearning.

The dev set is used to tune parameters, _____ features, and make other decisions about the learning algorithm.

Match each dev set use from Machine Learning Yearning to the activity that exemplifies it.

Arrange the stages of a machine learning project that involve the dev set in the correct workflow order.

Question:
Describe the main functions of a development set in a machine learning project and provide the alternative name sometimes used for it.

Sample answer:
The development set, sometimes called the hold-out cross validation set, is primarily used to evaluate models so that the team can tune parameters, select features, and make other decisions regarding the learning algorithm.

Key points:
- Mentions the alternative name: hold-out cross validation set.
- Identifies its use for tuning parameters.
- Identifies its use for selecting features.
- Notes it is used to make other decisions regarding the learning algorithm.

Rubric:
Award full credit if the student accurately provides the alternative name and lists the three main functions of the dev set.

Explain the role and alternative terminology for the development set.

Case context:
An engineering team is building a predictive algorithm. They have a full dataset and need to decide which data columns to use as inputs (features) and what specific configuration settings to apply to their algorithm (parameters).

Question:
Which specific data partition should the team evaluate their algorithm on to make these choices, and what is another name for this partition?

Sample answer:
The team should use the development set, which is also sometimes called the hold-out cross validation set, to tune their parameters and select their features.

Key points:
- Identifies the development set as the correct partition.
- Provides the alternative name: hold-out cross validation set.
- Connects the team's need to select features and tune parameters directly to the dev set's stated purpose.

Rubric:
The answer should correctly identify the development set and its alternative name as the proper tool for the team's parameter and feature selection tasks.

Determine the correct data partition for tuning a model's parameters and features.

Question:
According to the concept text, what are the three specific actions you use a development set to perform regarding the learning algorithm?

Sample answer:
A development set is used to tune parameters, select features, and make other decisions regarding the learning algorithm.

Key points:
- Tune parameters.
- Select features.
- Make other decisions regarding the learning algorithm.

Rubric:
The response must accurately list all three actions associated with the development set.

List the three primary uses of the development set.

A development set, sometimes called a hold-out cross validation set, is used to tune parameters, select features, and make other decisions about the learning algorithm.

Google

Machine learning is a field that studies how computer systems can utilize experience—typically in the form of data—to improve their performance on specific tasks. It integrates principles from statistics, data mining, and optimization, and is frequently employed as a primary method for implementing artificial intelligence solutions.

Machine Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Supervised learning learns a function mapping inputs x to outputs y using labeled training examples (x,y). Linear regression, logistic regression, and neural networks are examples of supervised learning algorithms.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

Development Set (Dev Set)

A test set is used to evaluate algorithm performance, but not to make decisions about which learning algorithm or parameters to use.

Test Set

When multiple evaluation metrics matter, such as accuracy and running time of a learning algorithm, one way to combine them is to define an acceptable value for one metric and then optimize another metric subject to meeting that criterion. In the passage's example, running time is the satisficing metric and accuracy is the optimizing metric.

Optimizing and Satisficing Metrics

For a new machine learning problem, it is very difficult to know in advance which approach will work best. Even experienced machine learning researchers usually try out many dozens of ideas before they discover something satisfactory.

Difficulty of Knowing the Best ML Approach in Advance

Machine learning is a highly iterative process. When building a machine learning system, the author often follows a loop: start with an idea about how to build the system, implement the idea in code, and carry out an experiment that tells how well the idea worked. Based on what is learned, generate more ideas and keep iterating; many dozens of ideas may be tried before finding one that is satisfactory.

Iterative Loop of Machine Learning Development

There are two major sources of error in machine learning: bias and variance. Understanding them helps decide whether adding data and other tactics to improve performance are a good use of time.

Bias and Variance as Two Major Sources of Error

End-to-end deep learning can directly learn outputs that are more complex than a number. With the right labeled input-output pairs, end-to-end learning can sometimes learn outputs such as a sentence, an image, audio, or other outputs richer than a single number.

End-to-End Learning

A pipeline may be flawed if the overall system falls far short of human-level performance even though each individual component is at or near human-level performance given its inputs. This suggests that the pipeline is missing information and should be redesigned.

Machine Learning Pipeline System

Sentiment classification is the problem of recognizing whether text expresses a positive or negative opinion, such as whether an online product review says the writer liked or disliked the product.

Sentiment Classification

Machine learning strategy is the practice of choosing useful directions for improving a machine learning application. It uses clues from the problem to decide what is useful to try and what is not useful to try.

Machine Learning Strategy

Recent progress in deep learning has been driven by data availability and computational scale.

Learn Before

Related

Learn After