Case Study

Case Study: Updating evaluation data after a product launch

Case context: A team has shipped an online recommendation service. After launch, it starts collecting real user interactions that better reflect the live environment than the older collected data used to build its dev and test sets.

Question: What can the team do with this newly available information?

Sample answer: The team can refresh both the development set and the test set using the real post-launch user data.

Key points:

  • The product is already in use
  • The new data comes from actual users after launch
  • Both the development set and the test set can be updated

Rubric: Full credit requires mentioning the post-launch user data as the new source and both dev and test sets as the datasets to update. Do not award credit for unsupported extra steps or claims.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI