Choosing threshold metrics and a best-score metric for an on-device spam filter
Context: A product team is building an on-device spam filter for a messaging app. The team is comparing candidate models on three criteria: app bundle size, message-processing latency on a midrange phone, and classification accuracy. The team needs a clear rule for deciding which model to ship.
Question: Using the N-1 satisficing metrics and one optimizing metric framework, explain which two criteria should be treated as constraints and which one should be maximized. State the rule for selecting the final model.
Model answer: Treat bundle size and latency as the satisficing metrics. Set acceptable thresholds for both—for example, the model must fit within the storage budget and respond fast enough for a good user experience. Treat accuracy as the optimizing metric. First remove any model that fails either threshold; then, from the remaining models, choose the one with the highest accuracy.
Key points:
- Bundle size and latency are the criteria that must pass their thresholds.
- Accuracy is the metric to maximize.
- The final model is the highest-accuracy option among those that satisfy both constraints.
Rubric: Award full credit only if the response: 1) identifies bundle size and latency as the satisficing metrics; 2) identifies accuracy as the optimizing metric; 3) states the two-stage selection rule: filter by the thresholds first, then choose the highest-accuracy model from the survivors.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
A model evaluation checklist has one headline score and several minimum standards. If there are N standards in total, how many should be treated as satisficing metrics?
A satisficing metric is evaluated by whether it reaches an acceptable target, not by how large it can become.
The last metric in an N-1 tradeoff setup
If a decision process uses N metrics, how many should usually be satisficing metrics?
A satisficing metric is judged by whether it reaches a preset threshold rather than by being improved without limit.
When comparing N criteria, _____ are usually assigned as satisficing metrics.
Match each term to its correct description in a prioritization framework.
Put the steps for setting up one optimizing metric and several satisficing metrics in order.
Why is app package size often treated as an important constraint for smartphone applications?
In this framework, it is acceptable to choose several optimizing metrics at the same time.
Which Metrics Are Satisficing?
Match each criterion from the delivery-app example to its role in the framework.
Put the evaluation steps in order for selecting one model under hard constraints and a single score to maximize.
Why do threshold checks and one score make multi-criterion model selection easier?
Choosing threshold metrics and a best-score metric for an on-device spam filter
What does it mean to use a satisficing metric?