University of Michigan - Ann Arbor

If you find that evaluation metric is not giving the correct rank order preference for what is actually better algorithm, then there's a time to think about defining a new evaluation metric. 

Even if you can't define the perfect evaluation metric and dev set, just set something up quickly and use that to drive the speed of your team iterating. And if later down the line you find out that it wasn't a good one or you have better idea, change it at that time.

When to change dev/test sets and metrics

What to do if: evaluation reference is not working

What to do if: you are doing well on metric but not on dev/test set


A recommended workflow for starting a new machine learning project is to:
1. Set up a development/test set and an evaluation metric.
2. Build an initial basic system as quickly as possible.
3. Use bias/variance analysis and error analysis to prioritize next steps for iterative improvement.

Learn Before

Related