Google

For a pipeline with steps A, B, and C, test each dev-set mistake by manually replacing A with a perfect output and running the rest of the pipeline. If the system becomes correct, attribute the error to A; otherwise repeat with B, and if that still does not fix the output, attribute the error to C.

General Error Attribution Procedure for Multi-Step Pipelines

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

For a self-driving car pipeline, a closed-track scenario where the car chooses a more jarring steering direction than a skilled driver can be analyzed by manually perfecting the detect-cars output, then the detect-pedestrians output, and finally attributing remaining errors to path planning.

Self-Driving Car Error Attribution Scenario

For pipeline error attribution, ML pipeline components should follow a directed acyclic graph ordering: components can be computed in a fixed left-to-right order, and later components should depend only on earlier components outputs. Any A-to-B-to-C mapping that follows this ordering can still give useful guidance.

DAG Ordering for Pipeline Error Attribution

What is the purpose of manually replacing a component's output with a 'perfect' output in the error attribution procedure?

If replacing A's output with a perfect output causes the pipeline to produce a correct result, the error should be attributed to component B.

In a pipeline A → B → C, if replacing A's output with a perfect output fixes the system, you attribute the error to _____.

Match each error attribution scenario to the correct conclusion about which pipeline component caused the error.

Order the steps of the general error attribution procedure for a three-step pipeline A → B → C.

In a pipeline A → B → C, under what condition does the error attribution procedure conclude that component C caused the error?

The error attribution procedure requires retraining every pipeline component from scratch for each dev-set mistake analyzed.

The error attribution procedure is applied to each _____ the system makes on the dev set, one at a time.

Match each pipeline component in A → B → C to the diagnostic action taken when testing whether it is the source of a dev-set error.

Order the reasoning steps for deciding which component to prioritize for improvement after running error attribution across many dev-set mistakes.

Question:
In a multi-step pipeline containing steps A, B, and C, explain the systematic procedure for determining which component is responsible for a specific mistake made on the dev set.

Sample answer:
To determine which component is responsible for a dev-set mistake, you start by modifying component A's output to be a "perfect" output and running the rest of the pipeline (B and C). If the pipeline produces the correct overall output, the error is attributed to A. If not, you proceed to component B, manually feeding a perfect output for B into component C. If this fixes the output, the error is attributed to B. If the overall output is still incorrect after perfect inputs to B, the error is attributed to component C.

Key points:
- Modify component A's output to be "perfect" and run B and C.
- If the output is correct, attribute the error to A.
- If incorrect, modify component B's output to be "perfect" and run C.
- If the output is correct, attribute the error to B; otherwise, attribute it to C.

Rubric:
The response should correctly outline the sequential process of substituting perfect outputs starting from the first component. It must mention testing A, then B, and attributing the error to C if neither A nor B caused the mistake.

Describe the error attribution process for a three-step pipeline.

Case context:
A self-driving car uses a pipeline with three steps: Car Detection (A), Pedestrian Detection (B), and Path Planning (C). On a specific dev-set example, the car makes a poor path planning decision. You provide a "perfect" bounding box for Car Detection (A), but the final path is still incorrect. You then provide a "perfect" output for Pedestrian Detection (B), and the final path becomes correct.

Question:
Based on the general error attribution procedure, what should the engineer diagnose as the cause of the poor path planning decision for this specific dev-set mistake, and why?

Sample answer:
The engineer should diagnose Pedestrian Detection (B) as the cause of the error. The procedure dictates that if replacing A's output with a perfect one does not fix the system, you move to B. Since providing a perfect output for B resulted in a correct final path, it shows that if B had produced a better output, the overall system would have been correct. Therefore, the error is attributed to B.

Key points:
- Component B (Pedestrian Detection) is the source of the error.
- Providing perfect input to A did not fix the problem.
- Providing perfect input to B fixed the overall system output.

Rubric:
The response must identify component B (Pedestrian Detection) as the source of the error and correctly justify it using the logic of the error attribution procedure.

Diagnosing a self-driving car pipeline.

Question:
In a three-step pipeline A → B → C, if replacing both A's output and B's output with perfect outputs fails to correct a dev-set mistake, to which component do you attribute the error?

Sample answer:
You attribute the error to component C.

Key points:
- Component C is assigned the error when providing perfect inputs for preceding components A and B does not fix the overall output.

Rubric:
The response must explicitly state component C.

Learn Before

Related