Google

The Precision of a cat classifier is the fraction of images in the dev or test set that it labeled as cats that really are cats.

Precision (Cat Classifier)

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

There is often a tradeoff between having high precision and high recall.

Tradeoff Between Precision and Recall

What does Precision measure for a cat classifier?

True or False: Precision is computed only from images labeled as cats.

Precision is the fraction of images labeled as cats that _____.

Match each term to its description related to Precision.

Order the steps to compute Precision for a cat classifier.

Question:
Explain, using the definition of Precision, why relying on Precision by itself might give an incomplete picture of a cat classifier's performance on the dev or test set.

Sample answer:
Precision only measures the fraction of images labeled as cats that really are cats. A classifier could achieve high Precision by being very conservative and only labeling images as cats when it is extremely confident, even if this means it misses many actual cat images. Because Precision ignores what happens to actual cat images that were not labeled as cats, a classifier can score well on Precision while still performing poorly overall, which is why Machine Learning Yearning pairs it with Recall.

Key points:
- Precision is the fraction of labeled-cat images that really are cats
- Precision ignores actual cat images that were not labeled as cats
- A classifier can achieve high Precision by being overly conservative
- This one-sidedness motivates considering Recall alongside Precision

Rubric:
Full credit requires stating the definition of Precision, explaining that it only considers predicted-cat images, and noting that this ignores missed cat images, motivating the need for a complementary metric like Recall.

Explain why Precision alone can be misleading when evaluating a cat classifier.

Case context:
A team tests their cat classifier on the dev set. The classifier labels 100 images as cats. When the team checks these 100 images against the true labels, they find that 95 of them are actually cats and 5 are not cats (they were mislabeled as cats).

Question:
Based on the definition of Precision, what is the Precision of this classifier on this dev set, and what does this value indicate about the classifier's labeled-cat predictions?

Sample answer:
Precision is the fraction of images labeled as cats that really are cats, so Precision = 95/100 = 0.95, or 95%. This means that when the classifier labels an image as a cat, it is correct 95% of the time, indicating that most of the classifier's positive (cat) predictions are trustworthy, though it says nothing about cat images the classifier may have missed.

Key points:
- Precision = (labeled-cat images that really are cats) / (total images labeled as cats)
- Precision = 95/100 = 0.95 or 95%
- High Precision means labeled-cat predictions are mostly correct
- Precision alone does not indicate how many actual cats were missed

Rubric:
Full credit requires correctly computing Precision as 95/100 and correctly interpreting that it reflects the reliability of the classifier's cat-labeled predictions, not its coverage of all actual cats.

Diagnose a cat classifier's Precision from dev set results.

Question:
In one to three sentences, state what set of images forms the denominator when calculating the Precision of a cat classifier.

Sample answer:
The denominator is the total number of images in the dev or test set that the classifier labeled as cats, regardless of whether that label was correct.

Key points:
- Denominator is images labeled as cats by the classifier
- Not all images in the dev or test set
- Includes both correctly and incorrectly labeled cat images

Rubric:
Full credit for identifying the denominator as all images labeled as cats by the classifier, not all images in the dataset.

Learn Before

Related