Case context: A machine learning team has just been assigned a brand new application with no existing models or prior dev/test sets. Some team members argue for spending three weeks carefully curating a comprehensive, high-quality dev/test set and metric before writing any model code, believing this thoroughness will pay off later.

Question: Based on the guidance for new applications, should this team spend three weeks perfecting their initial dev/test set before starting development? What would you recommend instead, and why?

Sample answer: No, three weeks is too long for a brand new application. The guidance is to create an initial dev/test set and metric in less than one week, even if imperfect, so the team has a well-defined target and can start making progress quickly. Overthinking the initial dev/test set delays useful iteration; it is better to move fast now and refine the dev/test sets later, once the application matures, at which point spending more time (even months) becomes reasonable.

Key points:

• Three weeks contradicts the under-one-week guideline for new applications
• Imperfect but quick dev/test sets are preferable to prolonged perfecting
• Quick sets provide a well-defined target for the team
• Longer refinement (months) is appropriate later, for mature systems
• Overthinking early setup delays productive iteration

Rubric: Full credit recommends against the three-week delay, cites the under-one-week guideline for new applications, and notes that refinement can happen later once the system matures. Partial credit identifies the timeline issue without explaining the rationale.