What to do when the evaluation signal points the team in the wrong direction
Question: If the development or test setup stops reflecting what the team actually wants to optimize, what should the team do?
Sample answer: The team should revise the development/test sets or the evaluation metric and clearly communicate the new target to everyone involved.
Key points:
- Revise the development/test sets or the evaluation metric.
- Make sure the team understands the updated target.
Rubric: To earn full credit, the response must say that the evaluation setup or metric should be changed and that the new direction should be communicated to the team.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What should a team do when its validation metric no longer reflects the real business goal?
True or False: In a machine learning project, the validation split and evaluation metric must stay fixed from start to finish.
A fraud-detection team wants to compare many model tweaks. Why is it useful to lock in one validation set and one metric before the search begins?
Why is it useful to define an evaluation metric and a development set early in a machine learning project?
True or False: Once an ML project starts, changing the validation data or success metric is unusual and should generally be avoided.
A stable development set and metric let you _____ faster.
Match each project-management situation to its description in an ML workflow.
Put the evaluation update steps in order.
What should you do if your metric stops reflecting the real goal?
True or False: If your evaluation metric no longer reflects the real objective of the project, it is reasonable to revise the metric and update the team on the new direction.
When the validation set or metric no longer matches the real business goal, you should _____ them and tell the team about the new direction.
Match each evaluation practice with the principle it demonstrates.
Order the steps that reflect a practical approach to dev/test sets and evaluation metrics over a project’s life cycle.
Setting and Revising Evaluation Targets
Updating Evaluation Criteria When Product Goals Shift
What to do when the evaluation signal points the team in the wrong direction