One reliable way to improve an algorithm's performance today is to train a bigger network and get more data.

Bigger Networks and More Data as Reliable Improvement Levers

With one million training examples, Machine Learning Yearning favors using a neural network.

One Million Examples Can Favor a Neural Network

Achieving best performance with neural networks

Impact of network size on performance

The two keys to _____ performance

Matching scale components to their effects

Steps to maximize neural network performance

Question:
Explain why simply having a very large neural network is not sufficient for achieving the 'best performance'. What is the other necessary component and how do they interact?

Sample answer:
While training a very large neural network provides the capacity to learn highly complex functions, it requires a huge amount of data to achieve the best performance. Without huge data, the large network cannot reach its full potential. The two factors work together: the huge data provides the extensive examples necessary for the large network to learn effectively, allowing the combined system to reach the highest performance levels.

Key points:
- A very large neural network is required.
- A huge amount of data is the second necessary component.
- Both are required simultaneously for the 'best performance'.

Rubric:
The response must explicitly mention the need for a 'huge amount of data' alongside a large neural network. It should explain that the two components must be combined to achieve the best performance.

Analyzing the relationship between model size and data volume

Case context:
A team has built a speech recognition system using a moderately sized neural network and a dataset of 10,000 hours of audio. The performance has plateaued. They have the budget to either significantly increase the size of their neural network or collect 100,000 more hours of audio, but they cannot do both immediately.

Question:
Based on the principle that the best performance comes from having both a very large network and huge data, what should the team diagnose about their ultimate goal, even if they must take steps sequentially?

Sample answer:
The team should diagnose that achieving the absolute 'best performance' will ultimately require them to secure both the very large neural network and the huge amount of data. Scaling only one dimension will have limits, so their long-term roadmap must account for expanding both the network capacity and the dataset volume.

Key points:
- Diagnose that scaling only one factor has limits.
- Recognize the ultimate goal is scaling both dimensions.
- Best performance requires a very large network AND huge data.

Rubric:
Evaluates if the learner understands that scaling both dimensions is ultimately necessary for the best performance.

Scaling a speech recognition system

Question:
According to the text, what are the two specific conditions you must meet to obtain the best performance from a neural network?

Sample answer:
To obtain the best performance, you must train a very large neural network and have a huge amount of data.

Key points:
- Train a very large neural network.
- Have a huge amount of data.

Rubric:
The answer must list both conditions: a very large neural network and a huge amount of data.

The two requirements for top performance

The effect of training larger neural networks

Data requirements for large networks

Larger neural networks can obtain better performance, and the best performance comes from training a very large neural network while also having a huge amount of data.

Google

Recent progress in deep learning has been driven by data availability and computational scale.

Scale Drives Machine Learning Progress

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

Data availability is a major driver of recent deep learning progress because digital-device activity creates large amounts of data that can be fed to learning algorithms.

Data Availability as a Driver of Deep Learning Progress

Computational scale is a major driver of recent deep learning progress because it makes it possible to train neural networks large enough to take advantage of huge datasets.

Computational Scale as a Driver of Deep Learning Progress

Older learning algorithms such as logistic regression can plateau as more data is added, meaning their learning curves flatten and the algorithms stop improving even with more data.

Older Learning Algorithms Can Plateau with More Data

Large Neural Networks Benefit from Huge Data

In a small data regime, traditional algorithms may or may not do better depending on how features are hand-engineered. With only about 20 training examples, feature hand-engineering can matter more than choosing logistic regression or a neural network.

Learn Before

Related

Learn After