Google

When trading off N different criteria, such as a model's binary file size, running time, and accuracy, one approach is to set N-1 criteria as satisficing metrics that must meet certain values, and define the final criterion as the optimizing metric. For example, one can set acceptable thresholds for binary file size and running time, then optimize accuracy given those constraints.

Choosing N-1 Satisficing Metrics and One Optimizing Metric

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When trading off N criteria for an ML model, how many should be set as satisficing metrics?

A satisficing metric requires meeting a minimum threshold rather than optimizing the value as much as possible.

When N-1 criteria are set as satisficing metrics, the remaining criterion is called the _____ metric.

When balancing N criteria for a model, how many should be set as satisficing metrics?

Satisficing metrics must only meet a specified threshold value, rather than being maximized.

When trading off N different criteria, you set _____ of them as satisficing metrics.

Match each term to its correct description in the satisficing/optimizing framework.

Order the steps for setting up the N-1 satisficing / one optimizing metric framework for a model.

Why does Machine Learning Yearning cite binary file size as a criterion worth constraining for mobile apps?

In this framework, you may designate more than one criterion as the optimizing metric.

In the Machine Learning Yearning example, binary file size and running time serve as _____ metrics.

Match each criterion from the Machine Learning Yearning mobile-app example to its role in the framework.

Order the steps for evaluating a single candidate model using the satisficing/optimizing framework.

Question:
Analyze how designating N-1 criteria as satisficing metrics and only one as an optimizing metric facilitates model selection compared to trying to maximize all N criteria simultaneously. Draw from the examples and criteria described in the source text.

Sample answer:
When a team faces N different criteria (such as binary file size, running time, and accuracy), trying to optimize all of them at once makes it difficult to compare different models. The N-1 satisficing and 1 optimizing framework simplifies this. By setting threshold values for N-1 of the criteria (e.g., binary file size and running time must be under a certain limit), these criteria become binary constraints (pass/fail). Then, the team can focus solely on maximizing or minimizing the single optimizing metric (e.g., accuracy) among the models that meet all satisficing thresholds. This provides a clear, unambiguous rule for selecting the best model.

Key points:
- Setting threshold values for N-1 criteria transforms them into binary constraints.
- Designating a single optimizing metric provides a clear rule for model comparison.
- Helps navigate trade-offs among N criteria like binary file size, running time, and accuracy.

Rubric:
To earn full credit, the answer must: 1) Explain how setting threshold values for N-1 criteria transforms them into binary constraints (satisficing metrics); 2) Describe how having a single optimizing metric provides a clear, unambiguous rule for comparing models; 3) Use the criteria of binary file size, running time, and accuracy from the text to illustrate the concept.

How does the satisficing and optimizing framework simplify decision-making when trading off N different model evaluation criteria?

Case context:
A machine learning team is developing an image classification model for a mobile app. The model needs to balance three criteria: binary file size (users do not want to download large apps), running time (inference latency), and classification accuracy. The team needs to set up a evaluation framework to decide which candidate model to deploy.

Question:
Applying the N-1 satisficing and one optimizing metric framework described in the text, explain how the team should define the roles of binary file size, running time, and accuracy, and state the exact rule they would use to select the best model.

Sample answer:
Under this framework, the team should designate N-1 (in this case, 2) criteria as satisficing metrics and the remaining 1 criterion as the optimizing metric. Since binary file size and running time are constraints (e.g., the file must be smaller than a set threshold so users will download it, and running time must be below a certain limit), these should be defined as the satisficing metrics. Accuracy should be defined as the single optimizing metric. The selection rule is: identify all candidate models that meet the thresholds for binary file size and running time, and from that subset, select the model with the highest accuracy.

Key points:
- Binary file size and running time serve as the satisficing metrics with acceptable thresholds.
- Accuracy serves as the single optimizing metric to be maximized.
- The best model is the one with the highest accuracy among those meeting the file size and running time thresholds.

Rubric:
To earn full credit, the answer must: 1) Correctly identify binary file size and running time as the satisficing metrics that must meet a threshold; 2) Correctly identify accuracy as the optimizing metric; 3) Formulate the model selection rule of filtering by satisficing thresholds first, then maximizing accuracy.

Defining satisficing and optimizing metrics for a mobile machine learning application

Question:
According to the source text, what does it mean to set a criterion, such as a model's running time or binary file size, as a satisficing metric?

Sample answer:
Setting a criterion as a satisficing metric means that you simply require it to meet a certain threshold or acceptable value, rather than trying to maximize or minimize it.

Key points:
- Satisficing metrics simply require meeting a certain threshold value.
- They are not optimized or maximized; they only act as acceptable limits.

Rubric:
To earn full credit, the answer must explain that a satisficing metric only requires the model to meet a certain specified threshold value rather than being optimized or maximized.

Learn Before

Related