A speech recognition pipeline can compute hand-designed MFCC features, recognize phonemes, and then use a final recognizer to turn the phoneme sequence into a transcript.

Speech Recognition Pipeline with MFCCs and Phonemes

In the autonomous-driving pipeline, if car detection, pedestrian detection, and path planning are each near human-level given their inputs but the overall car performs far below humans using camera images, the pipeline is missing information. Adding a lane-marking detector can provide important missing information without making path planning consume raw images directly.

Autonomous Driving Pipeline Example

Hand-engineered components can limit system performance by throwing away information or forcing an imperfect intermediate representation. In the speech pipeline, MFCC features simplify the audio signal and phonemes are an imperfect representation of speech sounds.

Hand-Engineered Components Can Limit Performance

Hand-engineered components can simplify the learning problem and reduce how much data is needed. MFCC features can be robust to irrelevant properties such as speaker pitch, and reasonable phoneme representations can help the algorithm understand basic sound components.

Hand-Engineered Components Can Reduce Data Requirements

If one chooses not to use an end-to-end system, one must decide what steps the pipeline should contain and how those steps should connect.

Designing Non-End-to-End Pipelines

When choosing components for a non-end-to-end pipeline, one important factor is whether data can be collected easily to train each component.

Training Data Availability for Pipeline Component Selection

Component Learning in Pipeline Systems

Individual Component Success vs. Overall Performance

Training Data in Non-End-to-End _____ Design

Concepts of Machine Learning Pipeline Systems

Redesigning and Optimizing a Pipeline System

Question:
Describe the scenario in which a machine learning pipeline is considered flawed even if its individual components perform at or near human-level performance. Explain what this scenario implies about the pipeline design and what action must be taken.

Sample answer:
A pipeline is flawed if the overall system falls far short of human-level performance while each individual component performs at or near human-level given its inputs. This discrepancy implies that the pipeline is missing information between stages. To resolve this, the pipeline should be redesigned.

Key points:
- Overall system falls far short of human-level performance.
- Individual components perform at or near human-level given their inputs.
- The pipeline is missing information.
- The pipeline should be redesigned.

Rubric:
The response must explain that the flaw occurs when components succeed but the overall system fails, identify that the pipeline is missing information, and state that the pipeline must be redesigned.

Identifying and Resolving Missing Information in Pipelines

Case context:
You are building a pipeline for an autonomous vehicle. The pipeline contains a vehicle detection component and a path planning component that uses a non-learned algorithm. Individual testing shows that both components perform at human-level given their inputs. However, when integrated, the overall self-driving system falls far short of human-level driving performance.

Question:
Based on the performance of the components and the overall system, diagnose the problem with the pipeline and decide what action to take.

Sample answer:
The diagnosis is that the pipeline is flawed and missing information. Even though each component performs at human-level given its inputs, the overall system's poor performance reveals a lack of necessary information flow between components. The decided next step is to redesign the pipeline.

Key points:
- Diagnose that the pipeline is flawed.
- Identify that the pipeline is missing information.
- Propose to redesign the pipeline.

Rubric:
The response must diagnose the pipeline as flawed and missing information, and decide that the pipeline should be redesigned.

Case Study: Diagnosing an Autonomous Vehicle Pipeline

Question:
What is an example of a component in a pipeline that does not involve learning, according to the literature cited in the text?

Sample answer:
Algorithms from the literature on robot motion planning used for the final path planning step do not involve learning.

Key points:
- Robot motion planning algorithms.
- Used for the final path planning step.
- Do not involve learning.

Rubric:
The answer should identify robot motion planning or path planning algorithms as examples of non-learned components in the pipeline.

Robot Motion Planning in Pipelines

Pipeline Component Candidacy

Non-Linear Pipelines

A pipeline may be flawed if the overall system falls far short of human-level performance even though each individual component is at or near human-level performance given its inputs. This suggests that the pipeline is missing information and should be redesigned.

Google

Machine learning is a field that studies how computer systems can utilize experience—typically in the form of data—to improve their performance on specific tasks. It integrates principles from statistics, data mining, and optimization, and is frequently employed as a primary method for implementing artificial intelligence solutions.

Machine Learning

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Representational learning is a class of machine learning that focuses on automatically discovering the most appropriate way to represent data. By learning these representations directly from the data, it eliminates the need for manual feature engineering and allows models to effectively process and understand complex inputs.

Representational Learning

Supervised learning learns a function mapping inputs x to outputs y using labeled training examples (x,y). Linear regression, logistic regression, and neural networks are examples of supervised learning algorithms.

Supervised Learning

Kaggle is a popular online platform that hosts machine learning competitions. Each competition centers on a specific dataset, and many are sponsored by stakeholders who offer prizes for the winning solutions. The platform enables direct quantitative comparisons of competing approaches and fosters collaboration by allowing users to interact via forums and share code.

Kaggle Platform

Combining classical machine learning and deep learning within predictive analytics platforms enables users to forecast future trends and behaviors with high precision. These data-driven insights accelerate time-to-value across industries by optimizing courses of action and reducing margins of error, effectively replacing retrospective interpretations with forward-looking decision-making.

Predictive Analytics for Accelerated Decision-Making

A development set, sometimes called a hold-out cross validation set, is used to tune parameters, select features, and make other decisions about the learning algorithm.

Development Set (Dev Set)

A test set is used to evaluate algorithm performance, but not to make decisions about which learning algorithm or parameters to use.

Test Set

When multiple evaluation metrics matter, such as accuracy and running time of a learning algorithm, one way to combine them is to define an acceptable value for one metric and then optimize another metric subject to meeting that criterion. In the passage's example, running time is the satisficing metric and accuracy is the optimizing metric.

Optimizing and Satisficing Metrics

For a new machine learning problem, it is very difficult to know in advance which approach will work best. Even experienced machine learning researchers usually try out many dozens of ideas before they discover something satisfactory.

Difficulty of Knowing the Best ML Approach in Advance

Machine learning is a highly iterative process. When building a machine learning system, the author often follows a loop: start with an idea about how to build the system, implement the idea in code, and carry out an experiment that tells how well the idea worked. Based on what is learned, generate more ideas and keep iterating; many dozens of ideas may be tried before finding one that is satisfactory.

Iterative Loop of Machine Learning Development

There are two major sources of error in machine learning: bias and variance. Understanding them helps decide whether adding data and other tactics to improve performance are a good use of time.

Bias and Variance as Two Major Sources of Error

End-to-end deep learning can directly learn outputs that are more complex than a number. With the right labeled input-output pairs, end-to-end learning can sometimes learn outputs such as a sentence, an image, audio, or other outputs richer than a single number.

End-to-End Learning

Machine Learning Pipeline System

Sentiment classification is the problem of recognizing whether text expresses a positive or negative opinion, such as whether an online product review says the writer liked or disliked the product.

Sentiment Classification

Machine learning strategy is the practice of choosing useful directions for improving a machine learning application. It uses clues from the problem to decide what is useful to try and what is not useful to try.

Machine Learning Strategy

Recent progress in deep learning has been driven by data availability and computational scale.

Learn Before

Related

Learn After