Case Study

Diagnosing Slow Progress in a New Machine Learning Project

Case context: A machine learning team has started a new project. They are trying many different neural network architectures, but they haven't formalized their evaluation criteria or finalized their datasets yet. Consequently, their progress is extremely slow because they keep debating which architecture is actually performing better.

Question: Based on the principles of machine learning project structure, what is the first step this team should take to speed up their workflow, and what specific benefit will this provide?

Sample answer: The team should immediately establish an initial dev/test set and a single evaluation metric. The specific benefit of having these in place is that it will give them an objective way to measure performance, which will help the team iterate quickly instead of arguing over subjective results.

Key points:

  • The team currently lacks an objective evaluation mechanism.
  • They need to establish an initial dev/test set and metric.
  • This configuration will help them resolve debates and iterate quickly.

Rubric: The answer must identify the need to establish the dev/test sets and a metric, and state that this action will help the team iterate quickly.

0

1

Updated 2026-06-13

Contributors are:

Who are from:

Tags

Machine Learning

Deep Learning

Supervised Learning

Dive into Deep Learning @ D2L

Data Science

Machine Learning Strategy

Machine Learning Yearning @ DeepLearning.AI