A deep RNN uses multiple neuron layers per time step. This is, in essence, a three-dimensional neural network which progresses from left to right, from right to left, and from bottom to top. It doesn't matter if the model performs forward propagation from left to right or from right to left, but it needs to propagate information through time first and then through layers. This architecture provides higher learning capacity, but also requires more training data.  

Deep (Bidirectional) RNNs

An ordinary RNN is useful because the model takes in information about your output at a previous time step to predict your current and future outputs. Similarly, a Bidirectional RNN depends upon both outputs at previous and FUTURE outputs. This is useful in applications where your predictions in your model not only depend on the past, but the future. This would include things like handwriting recognition, speech recognition, and bioinformatics. 

Motivation for Bidirectional RNNs

The concept of using more than one RNN that have some type of relationship between the hidden units is expandable to any number of dimensions of inputs you may have. For example, in our 1 dimensional example of "Bidirectional RNNs", the proposal was to use two networks with hidden units feeding into each other (one net for the past and one for the future).

This concept can be used with any $n$ dimensional input model. In image recognition, four recursive neural networks can be used, two for the vertical axis and two for the horizontal axis.

Implementation of Bidirectional RNNs

A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

University of Michigan - Ann Arbor

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Bidirectional RNNs

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer. The initial layers of stacked networks can induce representations that serve as useful abstractions for further layers — representations that might prove difficult to induce in a single RNN. So it can outperform single-layer networks.

Stacked RNNs

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

NTMs(Neural Turing Machines) are neural network architectures that use external memory. These models are able to infer algorithmic tasks like copying, sorting, associative recall. According to the inventors of this architecture via NTMs they: "extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes"
These models are relatively new compared to LSTMs and RNNs. They were created in 2014. 

Neural Turing Machines (NTM)

https://arxiv.org/pdf/1410.5401.pdf?fbclid=IwAR1mft01ZW2dGNGyVczj4HR6E5H8xKpqTlniZeuI8JxGuSNzOdoNupq4j9M

Neural Turing Machines - Original Paper Reference

In the late 1990s, German researchers Hochreiter and Schmidhuber proposed the concept of Long Short-Term Memory (LSTM), which helps Recurrent Neural Networks (RNNs) retain information over extended sequences, rather than just between consecutive time steps. For example, a character’s name introduced at the beginning of a paragraph may be necessary to understand the context at the end. An LSTM-based RNN shares the same high-level architecture as a basic RNN (whether simple, bidirectional, or deep), but replaces standard activation functions with specialized LSTM cells.

Learn Before

Related

Learn After