Why do threshold checks and one score make multi-criterion model selection easier?
Question: Explain how treating several requirements as minimum acceptable levels, while reserving one measure for optimization, makes it easier to choose among candidate models than trying to improve every requirement at once. Use a concrete example.
Sample answer: When a project cares about several model properties at the same time, the choice can become hard to judge. For instance, a team building a spam filter might care about prediction quality, inference time, and memory use. If all three are treated as equal goals, one model may be fastest, another may be smallest, and a third may be most accurate, so there is no simple way to decide.
A more practical method is to set minimum acceptable limits for two of the requirements, such as keeping latency below 40 milliseconds and memory use below 150 MB. Those two requirements become pass-or-fail checks. Any model that fails either one is removed from consideration. Among the models that pass both checks, the team then compares only the remaining score, such as precision or F1, and selects the best one on that metric.
This approach works because it turns several hard-to-balance goals into a clear rule: first satisfy the required limits, then optimize a single score among the models that qualify.
Key points:
- Thresholds convert some requirements into binary pass/fail tests.
- One remaining metric can be used to rank only the models that pass the tests.
- This avoids trying to balance several conflicting objectives at the same time.
Rubric: To earn full credit, the answer must: 1) Explain that some criteria are turned into threshold-based constraints; 2) Describe how one metric is used to choose among the candidates that meet those constraints; 3) Include a concrete example with model criteria.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
A model evaluation checklist has one headline score and several minimum standards. If there are N standards in total, how many should be treated as satisficing metrics?
A satisficing metric is evaluated by whether it reaches an acceptable target, not by how large it can become.
The last metric in an N-1 tradeoff setup
If a decision process uses N metrics, how many should usually be satisficing metrics?
A satisficing metric is judged by whether it reaches a preset threshold rather than by being improved without limit.
When comparing N criteria, _____ are usually assigned as satisficing metrics.
Match each term to its correct description in a prioritization framework.
Put the steps for setting up one optimizing metric and several satisficing metrics in order.
Why is app package size often treated as an important constraint for smartphone applications?
In this framework, it is acceptable to choose several optimizing metrics at the same time.
Which Metrics Are Satisficing?
Match each criterion from the delivery-app example to its role in the framework.
Put the evaluation steps in order for selecting one model under hard constraints and a single score to maximize.
Why do threshold checks and one score make multi-criterion model selection easier?
Choosing threshold metrics and a best-score metric for an on-device spam filter
What does it mean to use a satisficing metric?