If a lot of training data is available for intermediate modules in a pipeline, a multi-stage pipeline is worth considering. This structure can be superior because the available data can be used to train those intermediate modules.

Intermediate Module Data Availability Favors Multi-Stage Pipelines

When choosing components for a non-end-to-end pipeline, one important factor is whether data can be collected easily to train each component.

Google

A pipeline may be flawed if the overall system falls far short of human-level performance even though each individual component is at or near human-level performance given its inputs. This suggests that the pipeline is missing information and should be redesigned.

Machine Learning Pipeline System

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

A speech recognition pipeline can compute hand-designed MFCC features, recognize phonemes, and then use a final recognizer to turn the phoneme sequence into a transcript.

Speech Recognition Pipeline with MFCCs and Phonemes

In the autonomous-driving pipeline, if car detection, pedestrian detection, and path planning are each near human-level given their inputs but the overall car performs far below humans using camera images, the pipeline is missing information. Adding a lane-marking detector can provide important missing information without making path planning consume raw images directly.

Autonomous Driving Pipeline Example

Hand-engineered components can limit system performance by throwing away information or forcing an imperfect intermediate representation. In the speech pipeline, MFCC features simplify the audio signal and phonemes are an imperfect representation of speech sounds.

Hand-Engineered Components Can Limit Performance

Hand-engineered components can simplify the learning problem and reduce how much data is needed. MFCC features can be robust to irrelevant properties such as speaker pitch, and reasonable phoneme representations can help the algorithm understand basic sound components.

Hand-Engineered Components Can Reduce Data Requirements

If one chooses not to use an end-to-end system, one must decide what steps the pipeline should contain and how those steps should connect.

Learn Before

Related

Learn After