Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)



One solution to gradient problems is called "clipping the gradient". There are different ways to accomplish this; element-wise clipping is one option, norm-clipping is another. Although these methods produce slightly different mathematical results, there is experimental evidence that both work similarly in practice. 

Clipping Gradients

The long short-term memory (LSTM) network is an extension to RNNs, in addressing problems such as the vanishing gradient problem and the inability of RNNs to carry forward critical information.

LSTMs divide the context management problem into two sub-problems: 
- Removing information no longer needed from the context
- Adding information likely to be needed for later decision making

Long Short-Term Memory (LSTM) Network

- Identity RNN with ReLU activation (solving vanishing gradient problem only)
- Gradient clipping
- Skip connections
- LSTM
- GRU

University of Michigan - Ann Arbor

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow. 

Vanishing/exploding gradient

Natural Language Processing with Sequence Models is the third course in the Natural Language Prcessing Specialization published by deeplearning.ai.

Coursera: Natural Language Processing with Sequence Models

A webpage explaining exploding gradients

https://machinelearningmastery.com/exploding-gradients-in-neural-networks/

A Gentle Introduction to Exploding Gradients in Neural Networks

Solutions for vanishing/exploding gradient

If we initialize $W$s as a matrix of 0s, then $\frac{d\mathcal L}{dz}$ in each neuron from each layer would get the same value and no matter how long you train the network, gradient descent would get stuck at the same point. This is usually called the symmetry breaking problem.

What happens if we initialize the weights of a feed forward network to 0s?

Exploding/vanshing gradients is a common problem in reccurrent neural networks. Consider a network where an input is multiplied by a matrix $\bold{W}$ $t$ times. Let $\bold{W}^t$ have eigendecomposition $\bold{V}\text{diag}(\bold{\lambda})^t\bold{V}^{-1}$.

We can see that if $\lambda>1$, our result will approach $\infty$ as $t$ gets large. This is an exploding gradient. Also, if $\lambda <1$, the result will approach $0$ as $t$ gets large. This is a vanishing gradient.

Learn Before

Related

Learn After