Using Hard Limits and One Score to Compare Models
When a model must satisfy several criteria, one practical strategy is to turn most of them into minimum requirements and reserve a single metric for optimization. For example, in a speech-recognition system you might require latency under 100 ms and RAM use under 250 MB, then choose the model with the highest recognition accuracy among the candidates that meet those limits. This approach narrows the search to acceptable models and then ranks them by the main objective.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Acceptable-Threshold Metric
Primary Optimization Metric
Using Hard Limits and One Score to Compare Models
Using Two Error Rates in a Voice-Trigger Goal
Shared Evaluation Metric Lets a Team Move Faster
Latency limit versus F1 score
What Counts as a Satisficing Metric
Combining Multiple Evaluation Criteria
Choosing the Right Evaluation Metric
Applying Satisficing and Optimizing Metrics
Explaining Optimizing and Satisficing Metrics
Choosing a deployment metric under a hard device limit
Why Latency Is a Satisficing Metric
What does a satisficing metric ask for?
When a system has both an optimizing metric and a satisficing metric, the best plan is to maximize the optimizing metric even if the satisficing metric is violated.
Learn After
A model evaluation checklist has one headline score and several minimum standards. If there are N standards in total, how many should be treated as satisficing metrics?
A satisficing metric is evaluated by whether it reaches an acceptable target, not by how large it can become.
The last metric in an N-1 tradeoff setup
If a decision process uses N metrics, how many should usually be satisficing metrics?
A satisficing metric is judged by whether it reaches a preset threshold rather than by being improved without limit.
When comparing N criteria, _____ are usually assigned as satisficing metrics.
Match each term to its correct description in a prioritization framework.
Put the steps for setting up one optimizing metric and several satisficing metrics in order.
Why is app package size often treated as an important constraint for smartphone applications?
In this framework, it is acceptable to choose several optimizing metrics at the same time.
Which Metrics Are Satisficing?
Match each criterion from the delivery-app example to its role in the framework.
Put the evaluation steps in order for selecting one model under hard constraints and a single score to maximize.
Why do threshold checks and one score make multi-criterion model selection easier?
Choosing threshold metrics and a best-score metric for an on-device spam filter
What does it mean to use a satisficing metric?