Case Study

Estimating Bias From Training Error and a Human Performance Baseline

Case context: You build a classifier that detects whether satellite photos show flood damage. Its training error is 9%. An experienced human review team, limited by ambiguous images, can do no better than 8% error on the same task.

Question: Given the training error and the best achievable human-level error, what should you conclude about bias, and should bias-reduction be a priority?

Sample answer: The best achievable error is 8%. Because the classifier's training error is 9%, the gap is only 1%. That means the model's avoidable bias appears small, so bias-reduction work should not be the main priority.

Key points:

  • The best achievable error is 8%.
  • The gap from training error to that level is only 1%.
  • The model has little remaining avoidable bias.
  • Bias-reduction changes are not the top priority.

Rubric: The learner should identify the 8% baseline, recognize the 1% gap, and conclude that bias-reducing changes should not be prioritized.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI