Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)



One solution to gradient problems is called "clipping the gradient". There are different ways to accomplish this; element-wise clipping is one option, norm-clipping is another. Although these methods produce slightly different mathematical results, there is experimental evidence that both work similarly in practice. 

Clipping Gradients

The long short-term memory (LSTM) network is an extension to RNNs, in addressing problems such as the vanishing gradient problem and the inability of RNNs to carry forward critical information.

LSTMs divide the context management problem into two sub-problems: 
- Removing information no longer needed from the context
- Adding information likely to be needed for later decision making

Long Short-Term Memory (LSTM) Network

- Identity RNN with ReLU activation (solving vanishing gradient problem only)
- Gradient clipping
- Skip connections
- LSTM
- GRU

University of Michigan - Ann Arbor

In a neural network with many time steps or layers, a gradient at the early layer is the product of all the terms from the later layers, which leads to an inherently unstable situation. Especially when the value of gradient has become so small, it no longer updates properly or is vanished eventually. Exploding gradient can be considered as the opposite of vanishing process. The updated weights using gradient descent become so large that they cause the whole network to become unstable, which leads to numerical overflow.

Vanishing/exploding gradient

Natural Language Processing with Sequence Models is the third course in the Natural Language Prcessing Specialization published by deeplearning.ai.

Coursera: Natural Language Processing with Sequence Models

A webpage explaining exploding gradients

https://machinelearningmastery.com/exploding-gradients-in-neural-networks/

A Gentle Introduction to Exploding Gradients in Neural Networks

Solutions for vanishing/exploding gradient

If we initialize the weight matrix $$W$$ as a matrix of zeros, then the gradient $$\frac{d\mathcal{L}}{dz}$$ in each neuron from each layer will receive the exact same value. No matter how long the network is trained, gradient descent will update these parameters identically, preventing the neurons from learning distinct features. This failure to differentiate neuron weights is known as the symmetry problem. Furthermore, because the weights are zero, the gradients propagated backward to earlier layers are multiplied by zero, immediately contributing to the vanishing gradient problem.

Zero Weight Initialization in Feed-Forward Networks

When gradients explode during neural network training, the gradient norm $$\|\mathbf{g}\|$$ becomes excessively large. In such worst-case scenarios, a single gradient step can undo the progress made over the course of thousands of training iterations. Consequently, training often diverges, entirely failing to reduce the value of the objective function. Even in cases where training eventually converges, the process remains highly unstable due to massive spikes in the loss.

Impact of Exploding Gradients on Model Training

When minimizing an objective function using the hyperbolic tangent ($$	anh$$) activation function, optimization can stall due to the vanishing gradient problem. For example, if an algorithm attempts to minimize $$f(x) = 	anh(x)$$ starting at $$x = 4$$, the gradient is extremely small. Since the derivative is $$f'(x) = 1 - 	anh^2(x)$$, the gradient evaluates to $$f'(4) = 0.0013$$. Consequently, the optimization process gets stuck and makes negligible progress for a long time. This severe saturation issue is one of the primary reasons training deep learning models was notoriously tricky before the widespread adoption of the ReLU activation function.

Vanishing Gradient of the Tanh Activation Function

When optimization in deep neural networks stalls—frequently a consequence of vanishing gradients—a common mitigation strategy is reparametrization. This approach involves altering the mathematical formulation of the problem to create a more favorable loss landscape, thereby allowing optimization algorithms to resume making progress.

Reparametrization to Mitigate Stalling Optimization

Vanishing and exploding gradients are common problems in recurrent neural networks. Consider a network where an input is multiplied by a weight matrix $$ \mathbf{W} $$ for $$ t $$ time steps. Let $$ \mathbf{W}^t $$ have the eigendecomposition $$ \mathbf{V} \Lambda^t \mathbf{V}^{-1} $$, where $$ \Lambda $$ is a diagonal matrix of eigenvalues. We can see that if an eigenvalue $$ \lambda > 1 $$, the result will approach $$ \infty $$ as $$ t $$ gets large, leading to an exploding gradient. Conversely, if $$ \lambda < 1 $$, the result will approach $$ 0 $$ as $$ t $$ gets large, resulting in a vanishing gradient.

Learn Before

Related

Learn After