Case context: A product team is training a speech-to-text model using simulated studio recordings. After testing, they observe three patterns: the training error is much worse than human performance, the training-dev error is noticeably worse than the training error, and the error on a held-out set of genuine customer calls is even worse than the training-dev error.

Question: Using these results, determine which of the three main error categories are present in the model: avoidable bias, variance, and data mismatch. Explain how each metric gap supports your conclusion.

Sample answer: This model shows all three problems: avoidable bias, variance, and data mismatch. The training error being far above human-level performance indicates avoidable bias, because the model is not fitting the training task well enough. The jump from training error to training-dev error indicates variance, since performance drops on data drawn from the same general distribution. The further increase from training-dev error to the customer-call test error indicates data mismatch, because the evaluation data differ from the simulated training data.

Key points:

• High avoidable bias is shown by the gap between human-level error and training error.
• High variance is shown by the gap between training error and training-dev error.
• Data mismatch is shown by the gap between training-dev error and the held-out real-data error.
• The correct diagnosis is that all three error sources are present at the same time.

Rubric: The response must identify avoidable bias, variance, and data mismatch as all present, and it must tie each one to the relevant performance gap in the scenario.