If we initialize $W$s as a matrix of 0s, then $\frac{d\mathcal L}{dz}$ in each neuron from each layer would get the same value and no matter how long you train the network, gradient descent would get stuck at the same point. This is usually called the symmetry breaking problem.

University of Michigan - Ann Arbor

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow. 

Vanishing/exploding gradient

Symmetry breaking refers to a requirement of initializing machine learning models like neural networks.

When a neural network model has weights, all initialized to the same value, it can be difficult or impossible for the weights to differ as the model is trained. This is known as the “symmetry” problem.

Initializing the model to small random values breaks the symmetry and allows different weights to learn independently of each other.

Symmetry Breaking in Deep Learning


- Identity RNN with ReLU activation (solving vanishing gradient problem only)
- Gradient clipping
- Skip connections
- LSTM
- GRU

Solutions for vanishing/exploding gradient

A webpage explaining exploding gradients

https://machinelearningmastery.com/exploding-gradients-in-neural-networks/

A Gentle Introduction to Exploding Gradients in Neural Networks

What happens if we initialize the weights of a feed forward network to 0s?

Exploding/vanshing gradients is a common problem in reccurrent neural networks. Consider a network where an input is multiplied by a matrix $\bold{W}$ $t$ times. Let $\bold{W}^t$ have eigendecomposition $\bold{V}\text{diag}(\bold{\lambda})^t\bold{V}^{-1}$.

We can see that if $\lambda>1$, our result will approach $\infty$ as $t$ gets large. This is an exploding gradient. Also, if $\lambda <1$, the result will approach $0$ as $t$ gets large. This is a vanishing gradient.

When are exploding/vanishing gradients a problem?

Suppose you have built a neural network. You decide to initialize the weights and biases to be zero. Which of the following statements is true?

If all parameters of a hidden layer are initialized to a constant, such as $$\mathbf{W}^{(1)} = c$$, every hidden unit will receive the same inputs and parameters, producing identical activations during forward propagation. Consequently, during backpropagation, the gradients of the output with respect to the parameters $$\mathbf{W}^{(1)}$$ will all take the exact same value. Because gradient-based algorithms like minibatch stochastic gradient descent update the parameters using these uniform gradients, all elements of $$\mathbf{W}^{(1)}$$ will continue to have identical values after every iteration. The hidden layer will thus behave as if it has only a single unit, failing to realize the network's expressive power.

Learn Before

Related