Case Study

Choosing threshold metrics and a best-score metric for an on-device spam filter

Context: A product team is building an on-device spam filter for a messaging app. The team is comparing candidate models on three criteria: app bundle size, message-processing latency on a midrange phone, and classification accuracy. The team needs a clear rule for deciding which model to ship.

Question: Using the N-1 satisficing metrics and one optimizing metric framework, explain which two criteria should be treated as constraints and which one should be maximized. State the rule for selecting the final model.

Model answer: Treat bundle size and latency as the satisficing metrics. Set acceptable thresholds for both—for example, the model must fit within the storage budget and respond fast enough for a good user experience. Treat accuracy as the optimizing metric. First remove any model that fails either threshold; then, from the remaining models, choose the one with the highest accuracy.

Key points:

  • Bundle size and latency are the criteria that must pass their thresholds.
  • Accuracy is the metric to maximize.
  • The final model is the highest-accuracy option among those that satisfy both constraints.

Rubric: Award full credit only if the response: 1) identifies bundle size and latency as the satisficing metrics; 2) identifies accuracy as the optimizing metric; 3) states the two-stage selection rule: filter by the thresholds first, then choose the highest-accuracy model from the survivors.

0

1

Updated 2026-08-12

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI