Google

In contemporary deep learning, having abundant training examples and very high-capacity networks often makes the usual tension between underfitting and overfitting less restrictive. It is often possible to lower bias by using a larger model and adjusting regularization, while adding more data can often lower variance without materially changing bias.

Large Models and More Data Can Weaken the Tradeoff

Which change is most likely to lower bias in a modern neural network that is underfitting?

Modern deep learning removes the bias-variance tradeoff entirely.

Reducing variance by adding _____

Match each deep learning strategy with its effect on bias or variance

Order the model-improvement steps when a classifier has both high bias and high variance

Question:
Explain how today’s access to large training sets and high-capacity neural networks has altered the classic bias-variance relationship. Include concrete ways to lower bias and lower variance separately.

Sample answer:
In older machine learning settings, reducing bias usually came with a rise in variance, and reducing variance often meant accepting more bias. Modern deep learning is less constrained by that pattern because very large data sets and large neural networks make the tradeoff less rigid. A practitioner can often cut bias by expanding the network and tuning regularization so the model remains well behaved. At the same time, variance can often be reduced on its own by adding more training examples, without necessarily harming the model’s fit.

Key points:
- Large data sets and large networks weaken the classic tradeoff.
- Bias can often be lowered by increasing model capacity.
- Regularization helps control variance when using larger models.
- Variance can often be lowered independently by adding data.

Rubric:
The essay should contrast the older bias-variance view with the modern deep learning setting, and it must mention using larger networks with appropriate regularization to reduce bias, as well as using more data to reduce variance.

Why larger data sets and bigger networks change the bias-variance picture

Case context:
You are training a large neural network to detect risky financial transactions. The model has unusually high error on the training set. You also have access to a large pool of unlabeled transaction records that could be labeled at a cost, and you can afford substantial computing time.

Question:
According to modern deep learning practice, what should you do first to address the high training error, and what should you do if the model later still performs poorly on the development set?

Sample answer:
First, treat the high training error as a sign of underfitting and increase the model’s capacity, while also revisiting the regularization strength so it is not too restrictive. After that, check the development set. If the development error remains high, the problem is now likely excess variance, so I would label more of the available transaction data and add it to training.

Key points:
- High training error indicates underfitting or bias.
- Increase model size and adjust regularization to reduce bias.
- High development error after that points to variance.
- More labeled training data is a good next step for variance.

Rubric:
The response must identify high training error as a bias/underfitting problem and high development error as a variance problem. It must recommend increasing model capacity and tuning regularization first, then adding more labeled data if development performance is still poor.

Improve a transaction-risk classifier

Question:
In a modern deep learning setting, what is the most direct way to lower variance without trying to change the model's bias?

Sample answer:
Collect more training examples.

Key points:
- More training data usually lowers variance.
- This step does not directly change bias.

Rubric:
The answer must identify collecting or adding more training data as the method.

Learn Before

Related