Redirecting an ML Team's Target Alignment Mid-Project
Case context: A machine learning team starts a project with an initial dev set and evaluation metric to ensure they can iterate rapidly. Six weeks into the project, the team realizes their metric and dev set no longer reflect the actual product goals or user needs, meaning their models are optimizing for the wrong target.
Question: Based on the principles in Machine Learning Yearning, how should the team handle this misalignment regarding their dev/test sets and metrics, and what final step is critical once they make this decision?
Sample answer: The team should not hesitate to change their dev/test sets and evaluation metrics since changing them mid-project is common and not a big deal when they no longer point the team in the right direction. Once the change is decided, the critical final step is to make sure the entire team knows about the new direction.
Key points:
- Change the misaligned dev/test sets or evaluation metrics.
- Understand that changing metrics/sets is a common and acceptable practice.
- Ensure the team is informed about the new direction.
Rubric: Grading should verify that the student: 1. Advises changing the dev/test sets or metrics since they are misaligned. 2. Identifies this action as a common practice that is 'not a big deal'. 3. Specifies that the team must be notified of the new direction.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Related
What is the recommended action when your dev/test sets or evaluation metric no longer point your team in the right direction?
True or False: Changing dev/test sets or evaluation metrics partway through a machine learning project is considered unusual and should be avoided.
Having an initial dev/test set and metric helps your team _____ quickly during a machine learning project.
What is the primary benefit of establishing an initial dev/test set and evaluation metric at the start of an ML project?
True or False: Changing dev/test sets or evaluation metrics partway through an ML project is considered a rare and problematic event.
Having an initial dev/test set and metric helps you _____ quickly.
Match each dev/test set management situation with its correct description from ML Yearning.
Arrange the steps in the correct order for responding when you discover your current dev/test sets no longer guide your project effectively.
Midway through a project, you find your evaluation metric is no longer pointing your team in the right direction. What does ML Yearning recommend?
True or False: According to ML Yearning, if dev/test sets or metrics no longer point the team in the right direction, changing them is not a big deal.
When the dev/test sets or metric no longer point the team in the right direction, ML Yearning says to _____ them and ensure the team knows the new direction.
Match each role or characteristic of dev/test sets and metrics with the ML Yearning principle it reflects.
Arrange these statements in the order that best reflects ML Yearning's overall philosophy on managing dev/test sets across a project's lifecycle.
Analyzing Iterative Benefits and Adaptations of Dev/Test Sets
Redirecting an ML Team's Target Alignment Mid-Project
ML Yearning's Recommendation for Misaligned Metrics