Question:
Describe the scenario in which a machine learning pipeline is considered flawed even if its individual components perform at or near human-level performance. Explain what this scenario implies about the pipeline design and what action must be taken.

Sample answer:
A pipeline is flawed if the overall system falls far short of human-level performance while each individual component performs at or near human-level given its inputs. This discrepancy implies that the pipeline is missing information between stages. To resolve this, the pipeline should be redesigned.

Key points:
- Overall system falls far short of human-level performance.
- Individual components perform at or near human-level given their inputs.
- The pipeline is missing information.
- The pipeline should be redesigned.

Rubric:
The response must explain that the flaw occurs when components succeed but the overall system fails, identify that the pipeline is missing information, and state that the pipeline must be redesigned.

Google

A pipeline may be flawed if the overall system falls far short of human-level performance even though each individual component is at or near human-level performance given its inputs. This suggests that the pipeline is missing information and should be redesigned.

Machine Learning Pipeline System

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A speech recognition pipeline can compute hand-designed MFCC features, recognize phonemes, and then use a final recognizer to turn the phoneme sequence into a transcript.

Speech Recognition Pipeline with MFCCs and Phonemes

In the autonomous-driving pipeline, if car detection, pedestrian detection, and path planning are each near human-level given their inputs but the overall car performs far below humans using camera images, the pipeline is missing information. Adding a lane-marking detector can provide important missing information without making path planning consume raw images directly.

Autonomous Driving Pipeline Example

Hand-engineered components can limit system performance by throwing away information or forcing an imperfect intermediate representation. In the speech pipeline, MFCC features simplify the audio signal and phonemes are an imperfect representation of speech sounds.

Hand-Engineered Components Can Limit Performance

Hand-engineered components can simplify the learning problem and reduce how much data is needed. MFCC features can be robust to irrelevant properties such as speaker pitch, and reasonable phoneme representations can help the algorithm understand basic sound components.

Hand-Engineered Components Can Reduce Data Requirements

If one chooses not to use an end-to-end system, one must decide what steps the pipeline should contain and how those steps should connect.

Designing Non-End-to-End Pipelines

When choosing components for a non-end-to-end pipeline, one important factor is whether data can be collected easily to train each component.

Training Data Availability for Pipeline Component Selection

Component Learning in Pipeline Systems

Individual Component Success vs. Overall Performance

Training Data in Non-End-to-End _____ Design

Concepts of Machine Learning Pipeline Systems

Redesigning and Optimizing a Pipeline System

Identifying and Resolving Missing Information in Pipelines

Case context:
You are building a pipeline for an autonomous vehicle. The pipeline contains a vehicle detection component and a path planning component that uses a non-learned algorithm. Individual testing shows that both components perform at human-level given their inputs. However, when integrated, the overall self-driving system falls far short of human-level driving performance.

Question:
Based on the performance of the components and the overall system, diagnose the problem with the pipeline and decide what action to take.

Sample answer:
The diagnosis is that the pipeline is flawed and missing information. Even though each component performs at human-level given its inputs, the overall system's poor performance reveals a lack of necessary information flow between components. The decided next step is to redesign the pipeline.

Key points:
- Diagnose that the pipeline is flawed.
- Identify that the pipeline is missing information.
- Propose to redesign the pipeline.

Rubric:
The response must diagnose the pipeline as flawed and missing information, and decide that the pipeline should be redesigned.

Case Study: Diagnosing an Autonomous Vehicle Pipeline

Question:
What is an example of a component in a pipeline that does not involve learning, according to the literature cited in the text?

Sample answer:
Algorithms from the literature on robot motion planning used for the final path planning step do not involve learning.

Key points:
- Robot motion planning algorithms.
- Used for the final path planning step.
- Do not involve learning.

Rubric:
The answer should identify robot motion planning or path planning algorithms as examples of non-learned components in the pipeline.

Learn Before

Related