Google

Once a mobile app is live, its development and test sets may be revised using real user data collected after launch.

Refreshing Evaluation Sets After a Product Launch

Refreshing evaluation sets after deployment

True or False: Evaluation sets must remain frozen forever after deployment.

After deployment, evaluation sets may be refreshed with actual _____ data.

Match the post-release data update components

Order the steps for refreshing evaluation data after launch

Question:
In a brief analytical response, explain what new option becomes available after a mobile app is released and identify the data source and the evaluation sets it can affect.

Sample answer:
Once the app is in users’ hands, the team can use real usage data to revise the development and evaluation sets. In other words, release makes it possible to refresh those sets with examples from actual users instead of keeping the original splits unchanged forever.

Key points:
- The app has already been released.
- Real user data becomes available.
- The development set can be revised.
- The evaluation/test set can be revised.

Rubric:
A strong response should mention the post-release timing, name real user data as the source for updates, and identify both the development set and the test set as changeable. Do not award credit for unrelated claims about model training procedures or performance guarantees.

How can post-launch user data change evaluation sets in an app project?

Case context:
A team has shipped an online recommendation service. After launch, it starts collecting real user interactions that better reflect the live environment than the older collected data used to build its dev and test sets.

Question:
What can the team do with this newly available information?

Sample answer:
The team can refresh both the development set and the test set using the real post-launch user data.

Key points:
- The product is already in use
- The new data comes from actual users after launch
- Both the development set and the test set can be updated

Rubric:
Full credit requires mentioning the post-launch user data as the new source and both dev and test sets as the datasets to update. Do not award credit for unsupported extra steps or claims.

Case Study: Updating evaluation data after a product launch

Question:
Answer in one to three sentences: After a product has shipped, when is it appropriate to use real user data to revise the dev and test sets, and which sets may be refreshed?

Sample answer:
Once the app is live and you have collected real user behavior, you may rebuild or update both the development set and the test set.

Key points:
- After launch
- Using real user data
- Updating both development and test sets

Rubric:
The answer must mention that the app is already launched, that the data comes from actual users, and that both the dev set and the test set can be updated.

Learn Before

Related