Case context: A team is building a transcribed-note classifier for a hospital. They have 300,000 generic voice notes from other departments and 18,000 cardiology notes, which match the deployment setting. They reserve 6,000 cardiology notes for dev/test. The remaining 12,000 cardiology notes are mixed with the large generic set for training. During training, they track performance on the cardiology examples in the training set and on a separate cardiology development set.

Question: If the model scores well on the cardiology examples included in training but much worse on the cardiology development set, what is the most likely diagnosis, and what should the team do next?

Sample answer: That pattern suggests the model has memorized the limited cardiology examples it saw during training rather than learning patterns that transfer to new cardiology notes. The result supports the idea that the project is constrained by too little data from the target distribution. The next step is to gather more cardiology notes so the training set better matches the deployment distribution.

Key points:

• The gap indicates overfitting to the target-distribution training examples.
• The data signal says the real bottleneck is insufficient target-distribution data.
• The practical response is to collect more examples from the same distribution as dev/test.

Rubric: The response must identify overfitting to the limited target-distribution training data and recommend collecting more data from that distribution.