Case Study

Diagnosing a Fraud Model with 12% Training Error and 27% Dev Error

Case context: A fraud detection model is trained on bank transaction records. It makes mistakes on 12% of the training examples, which you use as an estimate of bias. On the dev set, its error rises to 27%, so the estimated gap due to variance is 15%.

Question: Using standard bias-variance ideas, diagnose this model. Describe what its results on the training set and dev set suggest, and explain why calling it simply overfitting or simply underfitting is not precise.

Sample answer: The model shows both high bias and high variance. Its 12% training error means it is not fitting the training examples well enough, and the jump to 27% on dev data shows it generalizes poorly as well. Because both problems appear together, using only the labels "overfitting" or "underfitting" hides part of the diagnosis.

Key points:

  • Identify both high bias and high variance.
  • Connect the 12% training error to poor fit on the training set.
  • Connect the higher dev error to a generalization gap.
  • Note that both issues can occur at the same time, so a single label is incomplete.

Rubric: Evaluates whether the student correctly identifies both high bias and high variance, links each to the reported training and dev errors, and explains why a single overfitting/underfitting label is not fully accurate.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Machine Learning Strategy

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Yearning @ DeepLearning.AI