As a system improves, mislabeled dev/test examples can become more important because their fraction grows relative to the total set of errors. When mislabeled dev examples add significant error to accuracy estimates, improving dev-set label quality becomes worthwhile.

Growing Relative Impact of Mislabeled Dev Examples

Mislabeled dev set labels are worth fixing when the fraction of the dev set that is mislabeled impedes the ability to quickly evaluate algorithms and decide whether Algorithm A or B is better.

Google

During error analysis, one may notice that some examples in the dev set are mislabeled. Here, "mislabeled" means that the pictures were already mislabeled by a human labeler before the algorithm encountered them; that is, in an example $(x, y)$, the class label $y$ has an incorrect value. For example, some pictures that are not cats may be mislabeled as containing a cat, and vice versa.

Mislabeled Examples in the Dev Set

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If one suspects that the fraction of mislabeled images is significant, one should add a Mislabeled category to keep track of the fraction of examples that are mislabeled. The shown columns are Image, Dog, Great cat, Blurry, Mislabeled, and Comments.

Adding a Mislabeled Category to the Error Analysis Spreadsheet

When to Fix Mislabeled Dev Set Labels

Whatever process is used to fix dev set labels should also be applied to test set labels so that the dev and test sets continue to be drawn from the same distribution. Fixing them together helps avoid optimizing dev-set performance only to be judged later on a different test-set criterion.

Learn Before

Related

Learn After