Case context: A machine learning team is developing an image classification model for a mobile app. The model needs to balance three criteria: binary file size (users do not want to download large apps), running time (inference latency), and classification accuracy. The team needs to set up a evaluation framework to decide which candidate model to deploy.

Question: Applying the N-1 satisficing and one optimizing metric framework described in the text, explain how the team should define the roles of binary file size, running time, and accuracy, and state the exact rule they would use to select the best model.

Sample answer: Under this framework, the team should designate N-1 (in this case, 2) criteria as satisficing metrics and the remaining 1 criterion as the optimizing metric. Since binary file size and running time are constraints (e.g., the file must be smaller than a set threshold so users will download it, and running time must be below a certain limit), these should be defined as the satisficing metrics. Accuracy should be defined as the single optimizing metric. The selection rule is: identify all candidate models that meet the thresholds for binary file size and running time, and from that subset, select the model with the highest accuracy.

Key points:

• Binary file size and running time serve as the satisficing metrics with acceptable thresholds.
• Accuracy serves as the single optimizing metric to be maximized.
• The best model is the one with the highest accuracy among those meeting the file size and running time thresholds.

Rubric: To earn full credit, the answer must: 1) Correctly identify binary file size and running time as the satisficing metrics that must meet a threshold; 2) Correctly identify accuracy as the optimizing metric; 3) Formulate the model selection rule of filtering by satisficing thresholds first, then maximizing accuracy.