Updating Evaluation Criteria When Product Goals Shift
Case context: A data science team is training a model for customer support routing. They choose a validation split and a scoring rule early in the project so they can compare ideas efficiently. After several weeks, they learn that the product will be used in a different workflow than expected, and the current evaluation setup rewards behavior that is no longer the main priority.
Question: How should the team respond to the mismatch between their validation/test setup and the real objective, and what should happen immediately after they decide on the change?
Sample answer: They should revise the validation and test setup, along with the evaluation metric, so that the project is judged against the goal that actually matters. Updating these targets partway through a project is a normal and acceptable step when the original choice stops matching the task. After the change is approved, the whole team should be told about the new evaluation direction.
Key points:
- Replace the misaligned validation/test setup or evaluation metric.
- Treat the update as a normal project adjustment.
- Communicate the new direction to everyone on the team.
Rubric: Grading should verify that the student: 1. Recommends changing the validation/test setup or metric because it no longer matches the goal. 2. Recognizes that this kind of change is a common and reasonable project decision. 3. States that the team must be informed of the updated direction.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What should a team do when its validation metric no longer reflects the real business goal?
True or False: In a machine learning project, the validation split and evaluation metric must stay fixed from start to finish.
A fraud-detection team wants to compare many model tweaks. Why is it useful to lock in one validation set and one metric before the search begins?
Why is it useful to define an evaluation metric and a development set early in a machine learning project?
True or False: Once an ML project starts, changing the validation data or success metric is unusual and should generally be avoided.
A stable development set and metric let you _____ faster.
Match each project-management situation to its description in an ML workflow.
Put the evaluation update steps in order.
What should you do if your metric stops reflecting the real goal?
True or False: If your evaluation metric no longer reflects the real objective of the project, it is reasonable to revise the metric and update the team on the new direction.
When the validation set or metric no longer matches the real business goal, you should _____ them and tell the team about the new direction.
Match each evaluation practice with the principle it demonstrates.
Order the steps that reflect a practical approach to dev/test sets and evaluation metrics over a project’s life cycle.
Setting and Revising Evaluation Targets
Updating Evaluation Criteria When Product Goals Shift
What to do when the evaluation signal points the team in the wrong direction