What to do if: evaluation reference is not working

What to do if: you are doing well on metric but not on dev/test set


A recommended workflow for starting a new machine learning project is to:
1. Set up a development/test set and an evaluation metric.
2. Build an initial basic system as quickly as possible.
3. Use bias/variance analysis and error analysis to prioritize next steps for iterative improvement.

Building an Initial Machine Learning System Quickly and Iterating

If you find that evaluation metric is not giving the correct rank order preference for what is actually better algorithm, then there's a time to think about defining a new evaluation metric. 

Even if you can't define the perfect evaluation metric and dev set, just set something up quickly and use that to drive the speed of your team iterating. And if later down the line you find out that it wasn't a good one or you have better idea, change it at that time.

University of Michigan - Ann Arbor

Setting up the goal for a machine learning project involves establishing well-defined evaluation metrics and datasets based on key principles:

1. **Single-number evaluation metric**: Select a consistent metric for comparing different deep learning models.
2. **Satisficing and optimizing metrics**: You can have multiple satisficing metrics (thresholds to meet) but only one optimizing metric.
3. **Dev and test set distribution**: Both datasets must have the same distribution. Choose dev and test sets that reflect the data you expect to get in the future and consider important to do well on.
4. **Dev and test set sizes**: For large datasets (e.g., containing 1,000,000 samples), the training dataset can have 98% of the data, while the dev and test sets only have 1% each. This splitting method differs from traditional machine learning algorithms.

Learn Before

Related

Learn After