Case Study: Updating evaluation data after a product launch
Case context: A team has shipped an online recommendation service. After launch, it starts collecting real user interactions that better reflect the live environment than the older collected data used to build its dev and test sets.
Question: What can the team do with this newly available information?
Sample answer: The team can refresh both the development set and the test set using the real post-launch user data.
Key points:
- The product is already in use
- The new data comes from actual users after launch
- Both the development set and the test set can be updated
Rubric: Full credit requires mentioning the post-launch user data as the new source and both dev and test sets as the datasets to update. Do not award credit for unsupported extra steps or claims.
0
1
Tags
Machine Learning
Deep Learning
Machine Learning Strategy
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Yearning @ DeepLearning.AI
Related
Refreshing evaluation sets after deployment
True or False: Evaluation sets must remain frozen forever after deployment.
After deployment, evaluation sets may be refreshed with actual _____ data.
Match the post-release data update components
Order the steps for refreshing evaluation data after launch
How can post-launch user data change evaluation sets in an app project?
Case Study: Updating evaluation data after a product launch
When should post-launch user data be used to refresh dev and test sets?
A logistics app is launched and begins collecting live customer records. Which action is supported?
Real user logs can be used to refresh both evaluation sets after launch.