In neural network optimization, exploding gradients occur when the gradient norm $$\|\mathbf{g}\|$$ becomes excessively large. Because the parameter update is proportional to the gradient, a single step with an exploded gradient can cause massive changes to the model, potentially undoing the progress of thousands of previous iterations. This results in training that either diverges completely, failing to minimize the objective function, or exhibits extreme instability with massive spikes in the loss function.

Impact of Exploding Gradients on Model Training

While gradient clipping is an effective regularization technique to mitigate the issue of exploding gradients during neural network training, it is unable to address the problem of vanishing gradients. Clipping only bounds the maximum magnitude of the gradient to prevent excessively large updates, but it provides no mechanism to prevent the gradient from becoming infinitesimally small.

Limitation of Gradient Clipping



One solution to gradient problems is called "clipping the gradient". There are different ways to accomplish this; element-wise clipping is one option, norm-clipping is another. Although these methods produce slightly different mathematical results, there is experimental evidence that both work similarly in practice. 

University of Massachusetts at Amherst

Claude

- Identity RNN with ReLU activation (solving vanishing gradient problem only)
- Gradient clipping
- Skip connections
- LSTM
- GRU

Solutions for vanishing/exploding gradient

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

Dive into Deep Learning

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

Clipping Gradients

The long short-term memory (LSTM) network is an extension to RNNs, in addressing problems such as the vanishing gradient problem and the inability of RNNs to carry forward critical information.

LSTMs divide the context management problem into two sub-problems: 
- Removing information no longer needed from the context
- Adding information likely to be needed for later decision making

Learn Before

Related

Learn After