Case context: A development team is building a cat photo sharing application. Each time they train a new cat classifier, they have no formal validation dataset or evaluation metric set up. Instead, developers compile the new model into the app, install it on a test phone, and spend hours browsing photos to see if the model performs better. They want to improve this workflow.

Question: Based on Andrew Ng's guidelines, diagnose the primary limitation of this team's current evaluation process. What infrastructure should they establish, and how will it change their decision-making process regarding which model changes to pursue?

Sample answer: The team's current manual evaluation process is incredibly slow and prevents them from easily detecting small improvements. To fix this, they should establish a specific dev set and evaluation metric. Having a dev set and metric will allow them to quickly and quantitatively detect whether new ideas result in small or large improvements. This fast feedback loop will enable them to make rapid, data-driven decisions on which classifier ideas are worth refining and which ones they should discard, rather than relying on slow, subjective manual testing.

Key points:

• Diagnose manual testing as slow and incapable of efficiently detecting small improvements.
• Recommend establishing a specific dev set and evaluation metric.
• Explain that a dev set and metric allow quick detection of improvements to guide whether to refine or discard ideas.

Rubric: The response must diagnose the slowness and inability to detect small improvements of manual testing. It must recommend establishing a specific dev set and evaluation metric. It must explain that this setup enables quick detection of improvements (small/large) to decide which ideas to refine or discard.