AI Agents

The most useful error buckets are the ones that suggest a realistic improvement. For example, if a photo classifier often fails because product labels cover part of the object, that category is especially valuable when the team has a plausible way to remove the obstruction or make the model more robust to it. Even so, it can still be worthwhile to track some categories you cannot fix immediately, because the main purpose is to learn where the system is weakest and where effort is most likely to pay off.

Prioritize Actionable Error Categories While Tracking Others

Which error groups are most useful to add first?

You should only analyze an error category if you already know how to fix it.

Building Intuition on _____ Opportunities

Choosing Useful Error Groups

Adding a Newly Discovered Error Category to Analysis

Question:
Why is it useful to record error categories even when you do not yet know how to improve them?

Sample answer:
It is still worthwhile to track those categories because error analysis is meant to deepen your understanding of the system. Recording them shows how often they occur and how much they matter, which helps you see where the model is struggling. Even without an immediate fix, that information can guide later work toward the problems that are most important or most frequent.

Key points:
- Improves understanding of the model’s behavior
- Shows the frequency and impact of each error type
- Helps prioritize future work
- A category can be useful before a fix is known

Rubric:
A strong response should explain that tracking difficult categories helps build insight and identify where future effort should go, even if no direct solution is available yet.

Why Track Hard-to-Fix Error Categories?

Case context:
You are building a model that labels pictures of fruit. A noticeable share of the mistakes come from photos that were edited with a strong warm-color filter in a mobile app. You do not yet know the best correction method, but you think it may be possible to estimate the original colors.

Question:
Should you add a "photo filter effects" category to your mistake log? Why or why not, using the ideas behind organizing errors?

Sample answer:
Yes. A separate "photo filter effects" category makes sense because you have at least one concrete direction for improvement: recovering or approximating the unedited image. You also should not wait until the fix is fully proven before recording these cases, because keeping track of them helps you judge whether this pattern is important enough to prioritize.

Key points:
- Yes, the category should be added
- It is useful because there is a plausible improvement idea (restoring the original image)
- Recording it helps you decide whether it is worth more attention later

Rubric:
A correct response will say the category should be added and explain that the learner has an idea for improvement, and/or that logging the cases helps assess whether this is a promising area to focus on later.

Tracking Filtered Photo Errors

Question:
Which kinds of error categories are the most useful to include in your analysis?

Sample answer:
The most useful error categories are the ones for which you already have a concrete idea for improvement.

Key points:
- A category is especially helpful when you can think of a plausible fix for it.

Rubric:
The answer must state that helpful categories are those the practitioner already has an idea for improving.

Learn Before

Related

Related