Case context: A team building a chest X-ray triage tool evaluates two classifiers using overall accuracy. Model A scores slightly higher than model B. In pilot use on emergency-room images, model A sometimes fails to flag obvious pneumothorax cases, while model B catches them reliably.

Question: As the product lead, which model should be deployed, and what should happen to the evaluation metric?

Sample answer: The team should deploy model B, because missing a dangerous condition is a serious failure even if model A has higher accuracy overall. The metric should be changed so that missed critical findings are heavily penalized or disqualify a model, rather than allowing a high average score to hide this defect.

Key points:

• Model A has higher accuracy but misses some pneumothorax cases
• Model B does not show the same miss pattern
• The safer choice is model B, not model A
• The current metric misses an important product requirement
• The metric should be revised to punish or block this kind of error

Rubric: Full credit requires choosing model B (or rejecting model A) with a reason tied to the dangerous miss, and stating that the metric needs to reflect that safety requirement.