In principle, increasing neural-network size and training data without limit can perform well on many learning problems. In practice, very large models become slow to train, and the available supply of additional training data can be exhausted.

Google

A simple formula for addressing bias and variance is to increase model size when a system has high avoidable bias, and to add data to the training set when a system has high variance.

Simple Bias/Variance Remediation Formula

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When a learning system has high avoidable bias, increasing the model size can help reduce bias because it allows the model to fit the training set better. For neural networks, this can mean adding layers or neurons; if increasing model size raises variance, regularization will usually eliminate the variance increase.

Increasing Model Size to Reduce Avoidable Bias

Adding more training data is the simplest and most reliable way to address high variance, so long as one has access to significantly more data and enough computational power to process it. Adding training data can usually reduce variance without affecting bias.

Adding Training Data to Reduce Variance

Scaling Model Size and Training Data Has Practical Limits

When a learning algorithm suffers from high avoidable bias, useful techniques include increasing model size, modifying input features based on error-analysis insights, reducing or eliminating regularization, and modifying the model architecture to better suit the problem. Adding more training data is not a helpful high-bias technique because it helps variance but usually has no significant effect on bias.

Techniques for Reducing Avoidable Bias

When a learning algorithm suffers from high variance, useful techniques include adding more training data, adding regularization, adding early stopping, using feature selection, and sometimes modifying input features or model architecture. Decreasing model size can reduce variance, but should be used with caution because regularization usually gives better classification performance when computational cost is not the concern.

Learn Before

Related