Choosing a deployment metric under a hard device limit
Case context: You are selecting a handwriting recognition model for a portable scanner. You want the highest possible recognition accuracy, but the scanner can only load models that use 120 MB of RAM or less.
Question: How should you treat accuracy and RAM usage when comparing candidate models? Specify which metric is optimizing and which is satisficing.
Sample answer: RAM usage should be treated as a satisficing metric because the scanner has a hard 120 MB limit. Any model above that limit is unacceptable, while any model at or below it is acceptable from the memory standpoint. Accuracy should be the optimizing metric. Among the models that meet the RAM limit, choose the one with the best accuracy.
Key points:
- RAM usage is the satisficing metric.
- The acceptable threshold is 120 MB or less.
- Accuracy is the optimizing metric.
- Select the most accurate model among those that satisfy the RAM limit.
Rubric: A correct response must identify the memory constraint as satisficing and accuracy as optimizing, with the decision rule of maximizing accuracy subject to the RAM cap.
0
1
Tags
D2L
Dive into Deep Learning @ D2L
Machine Learning
Deep Learning
Supervised Learning
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Acceptable-Threshold Metric
Primary Optimization Metric
Using Hard Limits and One Score to Compare Models
Using Two Error Rates in a Voice-Trigger Goal
Shared Evaluation Metric Lets a Team Move Faster
Latency limit versus F1 score
What Counts as a Satisficing Metric
Combining Multiple Evaluation Criteria
Choosing the Right Evaluation Metric
Applying Satisficing and Optimizing Metrics
Explaining Optimizing and Satisficing Metrics
Choosing a deployment metric under a hard device limit
Why Latency Is a Satisficing Metric
What does a satisficing metric ask for?
When a system has both an optimizing metric and a satisficing metric, the best plan is to maximize the optimizing metric even if the satisficing metric is violated.