Create the First Dev/Test Set and Metric Quickly
Question: How soon should a team define an initial dev/test set and evaluation metric when launching a brand-new machine learning project?
Sample answer: The team should set them up very early, ideally within the first week and often sooner.
Key points:
- Do this at the start of a new project
- Aim for less than one week
- This guidance is for new applications, not mature ones
Rubric: Full credit requires a clear statement that the initial dev/test set and metric should be created in under a week, or very early in the project. Partial credit gives an imprecise or incorrect timeframe.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
For a completely new product, about how quickly should a team set an initial metric and dev/test split?
True or False: The same short deadline that works for a brand-new project is also the right expectation for an older production system.
When a new project quickly defines dev and test sets, the team gets a clear _____ to work toward.
Match each project stage to the recommended pace for creating dev/test sets and a metric.
Put the steps in order for setting up dev/test sets and a metric for a new project quickly.
Why should a new machine-learning project define its evaluation setup quickly?
Should a new product team spend three weeks polishing its first evaluation set?
Create the First Dev/Test Set and Metric Quickly
Why should a team define dev and test sets early in a new project?
An initial validation set and evaluation metric on a new machine learning project can be rough at first.