Google

In modern deep learning settings, access to plentiful data and very large neural networks makes the bias-variance tradeoff less severe. One can often reduce bias without noticeably increasing variance by increasing network size and tuning regularization, and can often reduce variance without affecting bias by adding training data.

Modern Deep Learning Softens the Bias-Variance Tradeoff

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

How to reduce bias in modern deep learning

Deep learning completely eliminates the bias-variance tradeoff

Reducing variance by adding _____ to the model

Match the modern deep learning method to its effect on bias or variance

Order the steps to address high bias and then high variance sequentially

Question:
Explain how the availability of plentiful data and large neural networks has changed the traditional perspective on the bias-variance tradeoff. Give specific examples of how practitioners can manipulate bias and variance independently.

Sample answer:
In traditional machine learning, decreasing bias almost always increased variance, and vice versa. However, in modern deep learning, access to plentiful data and large neural networks softens this tradeoff. Practitioners can now often reduce bias without significantly increasing variance by increasing the network size and carefully tuning regularization. Conversely, they can reduce variance without hurting bias simply by adding more training data.

Key points:
- Plentiful data and large networks make the tradeoff less severe.
- Reduce bias by increasing network size.
- Use regularization to prevent variance increase when scaling the network.
- Reduce variance independently by adding training data.

Rubric:
The essay should clearly contrast the traditional tradeoff with the modern deep learning approach, explicitly mentioning the use of larger networks with tuned regularization to reduce bias, and adding data to reduce variance.

Explain how modern deep learning softens the bias-variance tradeoff

Case context:
You are training a large image recognition model. Your initial model shows a high error rate on the training set. You have access to a massive, unlabelled repository of images that can be labeled at a cost, as well as significant computational resources.

Question:
Based on modern deep learning principles, what two primary actions should you take in sequence to address the high training error, and then what should you do if the model subsequently performs poorly on the dev set?

Sample answer:
First, to address the high training error (high bias), I should increase the neural network size and carefully tune the regularization method. Once the bias is reduced without noticeably increasing variance, I should evaluate the dev set. If the dev set error is high (high variance), I should utilize the repository by labeling and adding more training data to the model.

Key points:
- Diagnose high training error as high bias.
- Diagnose high dev set error as high variance.
- Increase network size and tune regularization for bias.
- Add training data to resolve variance.

Rubric:
The response must correctly diagnose high training error as bias and high dev set error as variance. It must propose increasing network size and tuning regularization for the former, and adding training data for the latter.

Optimize a modern image recognition network

Question:
In a modern deep learning context, what is the primary method recommended to reduce variance without affecting bias?

Sample answer:
The primary method is to add more training data.

Key points:
- Add training data
- It reduces variance
- It does not affect bias

Rubric:
The answer must state that adding training data or gathering more data is the solution.

Learn Before

Related