Dev/Test Set Distribution Not Representative of Actual Distribution

Dev Set Overfitting from Repeated Evaluation

Metric Optimizes the Wrong Project Objective

Which scenario is a warning sign that your dev/test set or evaluation metric needs to change?

True or False: Discovering that your initial dev/test set or metric missed the mark is a serious setback that cannot be easily corrected.

If your _____ is no longer measuring what is most important to you, Ng recommends changing it rather than continuing to optimize for it.

Which of the following is the key warning sign that your dev/test set or evaluation metric may need to be changed?

Ng considers discovering that a dev/test set or metric missed the mark to be a serious setback that requires restarting the evaluation process from scratch.

If your metric is no longer measuring what is most important to your project, you should change the _____.

Match each cause of a dev set/metric incorrectly ranking classifiers to the fix Ng recommends.

Order the steps a team should take upon discovering their dev/test set or metric is no longer guiding them correctly.

Your dev set contains formal customer emails but users primarily submit short social media posts. Which cause does this best illustrate?

After changing your dev/test sets or evaluation metric, updating the project files is sufficient — there is no need to explicitly inform the team of the new direction.

If you have overfit to the dev set, Ng recommends getting more _____ data.

Match each problem scenario to the cause category it represents in Ng's framework for incorrect classifier ranking.

Order the reasoning steps for deciding whether and how to change an evaluation metric that may no longer reflect project goals.

Analyze the warning signs and causes of a development set incorrectly ranking classifiers

Diagnosing Classifier Ranking Mismatch in Spam Detection

Response to Overfitting the Development Set