A recommended workflow for starting a new machine learning project is to:
1. Set up a development/test set and an evaluation metric.
2. Build an initial basic system as quickly as possible.
3. Use bias/variance analysis and error analysis to prioritize next steps for iterative improvement.

University of Michigan - Ann Arbor

Google

If you find that evaluation metric is not giving the correct rank order preference for what is actually better algorithm, then there's a time to think about defining a new evaluation metric. 

Even if you can't define the perfect evaluation metric and dev set, just set something up quickly and use that to drive the speed of your team iterating. And if later down the line you find out that it wasn't a good one or you have better idea, change it at that time.

Learn Before

Related