Adding regularization, such as L2 regularization, L1 regularization, or dropout, reduces variance but increases bias.

Adding Regularization to Reduce Variance

Early stopping means stopping gradient descent early based on dev-set error. It reduces variance but increases bias, behaves much like regularization, and is sometimes called a regularization technique.

Early Stopping to Reduce Variance

Feature selection can reduce variance by decreasing the number or type of input features, but it can also increase bias. Small reductions in feature count are unlikely to have a large bias effect, while large reductions are more likely to matter, especially if too many useful features are excluded. In modern deep learning with plentiful data, practitioners are more likely to provide all features and let the algorithm learn which to use, but feature selection can still be useful when the training set is small.

Feature Selection for Variance Reduction

Decreasing model size, such as reducing neurons or layers, can decrease variance while increasing bias, but it is not generally recommended as a variance remedy. If computational cost is not the concern, adding regularization usually gives better classification performance; the main advantage of a smaller model is faster training.

Decreasing Model Size as a Variance Remedy

When a learning algorithm suffers from high variance, useful techniques include adding more training data, adding regularization, adding early stopping, using feature selection, and sometimes modifying input features or model architecture. Decreasing model size can reduce variance, but should be used with caution because regularization usually gives better classification performance when computational cost is not the concern.

Google

A simple formula for addressing bias and variance is to increase model size when a system has high avoidable bias, and to add data to the training set when a system has high variance.

Simple Bias/Variance Remediation Formula

Machine Learning Yearning, a free ebook from Andrew Ng, teaches you how to structure Machine Learning projects.
https://www.deeplearning.ai/machine-learning-yearning/

Note: The content of the book is aligned with the Coursera Deeplearning.ai specialization.  https://www.deeplearning.ai/deep-learning-specialization/ 

Machine Learning Yearning (Deeplearning.ai)

When a learning system has high avoidable bias, increasing the model size can help reduce bias because it allows the model to fit the training set better. For neural networks, this can mean adding layers or neurons; if increasing model size raises variance, regularization will usually eliminate the variance increase.

Increasing Model Size to Reduce Avoidable Bias

Adding more training data is the simplest and most reliable way to address high variance, so long as one has access to significantly more data and enough computational power to process it. Adding training data can usually reduce variance without affecting bias.

Adding Training Data to Reduce Variance

In principle, increasing neural-network size and training data without limit can perform well on many learning problems. In practice, very large models become slow to train, and the available supply of additional training data can be exhausted.

Scaling Model Size and Training Data Has Practical Limits

When a learning algorithm suffers from high avoidable bias, useful techniques include increasing model size, modifying input features based on error-analysis insights, reducing or eliminating regularization, and modifying the model architecture to better suit the problem. Adding more training data is not a helpful high-bias technique because it helps variance but usually has no significant effect on bias.

Learn Before

Related

Learn After