Setting and Revising Evaluation Targets
Question: Why do teams create an early development set and scoring rule, and what should they do if those choices stop matching the project objective?
Sample answer: An early development set and metric give the team a consistent reference for comparing experiments, which makes model work more efficient. If later results show that the chosen target rewards the wrong behavior or no longer reflects the real goal, the team should revise it instead of continuing to optimize the outdated measure. The updated target should then be communicated clearly so everyone works toward the same goal.
Key points:
- An early benchmark helps compare experiments efficiently.
- It is acceptable to revise that benchmark when it no longer matches the real objective.
- After revising it, the team should be told about the new standard.
Rubric: To receive full credit, the answer must: 1. Explain that early sets and metrics help the team iterate efficiently. 2. State that they may be changed when they no longer fit the project goal. 3. Note that the new target must be shared with the team.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
What should a team do when its validation metric no longer reflects the real business goal?
True or False: In a machine learning project, the validation split and evaluation metric must stay fixed from start to finish.
A fraud-detection team wants to compare many model tweaks. Why is it useful to lock in one validation set and one metric before the search begins?
Why is it useful to define an evaluation metric and a development set early in a machine learning project?
True or False: Once an ML project starts, changing the validation data or success metric is unusual and should generally be avoided.
A stable development set and metric let you _____ faster.
Match each project-management situation to its description in an ML workflow.
Put the evaluation update steps in order.
What should you do if your metric stops reflecting the real goal?
True or False: If your evaluation metric no longer reflects the real objective of the project, it is reasonable to revise the metric and update the team on the new direction.
When the validation set or metric no longer matches the real business goal, you should _____ them and tell the team about the new direction.
Match each evaluation practice with the principle it demonstrates.
Order the steps that reflect a practical approach to dev/test sets and evaluation metrics over a project’s life cycle.
Setting and Revising Evaluation Targets
Updating Evaluation Criteria When Product Goals Shift
What to do when the evaluation signal points the team in the wrong direction