Google

A pipeline can be debugged informally by comparing each component and the overall system to human-level performance. For path planning, the human comparison should use the same inputs as the component, such as only the earlier components outputs rather than raw camera images.

Comparing Pipeline Components to Human-Level Performance

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

If one pipeline component is far from human-level performance, that is a good reason to focus on improving that component.

Component Far from Human-Level Performance as Improvement Priority

If each individual component of an ML pipeline is at or near human-level performance but the overall pipeline falls far short of human-level performance, the pipeline is usually flawed and needs redesign.

Flawed Pipeline Despite Human-Level Components

When comparing the Plan path component to human-level performance in a self-driving pipeline, what inputs should the human evaluator receive?

ML Yearning describes comparing pipeline components to human-level performance as a rigorous, procedure-based debugging technique.

In informal pipeline debugging, you ask how _____ each component is from human-level performance.

Match each self-driving pipeline component from ML Yearning to its primary output or function.

Order the informal debugging questions ML Yearning recommends asking when evaluating a self-driving car pipeline.

Why does ML Yearning require the human evaluator for path planning to use only the detection components' outputs rather than camera images?

In ML Yearning's self-driving example, both the Detect cars and Detect pedestrians components feed their outputs directly into the Plan path component.

To compare the Plan path component fairly to a human, the human must plan the route using only the _____ of the detection components.

Match each informal debugging question to the pipeline element it targets in ML Yearning's self-driving car example.

Arrange the reasoning steps a practitioner follows when diagnosing a self-driving pipeline problem using human-level comparison.

Question:
When informally debugging a machine learning pipeline by comparing its components to human-level performance, why is it critical that the human evaluator is restricted to the exact inputs of the component being tested (e.g., outputs of prior components) rather than having access to the raw initial data (e.g., camera images)?

Sample answer:
It is critical because giving the human evaluator raw initial data (like camera images) instead of the component's actual inputs (like detection outputs) would result in an unfair comparison. If the human evaluates path planning using raw images, they are using information that the path planning component does not have access to. Restricting the human to the same inputs allows us to isolate the performance of the path planning component itself, rather than testing the combined effect of the detector errors and path planning errors.

Key points:
- Using raw inputs gives the human evaluator access to information not available to the component.
- Restricting the human to the same inputs ensures a fair, isolated comparison of that specific component.
- It prevents upstream component errors from confounding the evaluation of the downstream component.

Rubric:
Answers should explain that comparing a component to a human who has access to raw data mixes up errors from upstream components with errors of the current component. It must mention that using identical inputs isolates the performance of the specific component being evaluated.

Analyzing Pipeline Component Inputs for Human-Level Performance Comparison

Case context:
You are building a self-driving car pipeline. The car detection and pedestrian detection components output bounding boxes. The path planning component takes these bounding boxes and plans a trajectory. You want to informally debug the path planning component by comparing it to human-level performance. Your colleague proposes showing a human driver the raw front-facing camera videos to see if they would plan a better path.

Question:
Based on the principles of informal pipeline debugging, explain why your colleague's proposal is flawed and how you should structure the human comparison instead.

Sample answer:
My colleague's proposal is flawed because it allows the human driver to see raw camera videos. This gives the human access to rich visual features that the path planning component does not receive, such as road boundaries or un-detected obstacles. To structure the comparison correctly, the human should be given only the outputs of the car and pedestrian detection components (the bounding boxes/locations) and asked to plan a path using only that restricted information. This measures how well the planning component performs relative to a human given the same inputs.

Key points:
- Identifies that raw camera video provides extra information not available to the path planner.
- Explains that the human must plan a path using only the outputs from the detection components.
- States that the goal is to compare the component's performance to a human given identical inputs.

Rubric:
Learners must identify that showing raw video provides the human with more information than the component receives. They must specify that the correct setup involves providing the human only with the outputs from the detection components.

Evaluating a Self-Driving Path Planning Component

Question:
When comparing the performance of the 'Plan path' component to human-level performance, what specific constraint must be placed on the human's inputs to ensure a fair comparison?

Sample answer:
The human must plan the path using only the outputs of the previous components (car and pedestrian detection) rather than the raw camera images.

Key points:
- Human must only use outputs from the previous pipeline components (e.g., car/pedestrian detections).
- Human must not have access to the raw camera images.

Rubric:
The answer must specify that the human should only use the outputs of the preceding pipeline components (detection outputs) and not the raw inputs (camera images).

Learn Before

Related