To implement a multilayer Recurrent Neural Network (RNN) from scratch, the network can be constructed by instantiating each layer as an individual basic recurrent unit, such as an `RNNScratch` object, possessing its own independent set of learnable parameters. The initial layer processes the original sequence data, whereas every subsequent layer receives the hidden representations produced by the layer immediately preceding it.

Stacked RNN Implementation from Scratch

Implementing multiple layers of a Recurrent Neural Network (RNN) from scratch involves handling numerous logistical details, but modern deep learning frameworks abstract these away through high-level APIs. A deep architecture can be concisely defined by utilizing built-in recurrent functionalities and explicitly specifying a nontrivial number of hidden layers (e.g., setting a num_layers parameter) rather than relying on the default single-layer configuration. This approach generalizes single-layer implementations, allowing developers to easily instantiate deep models like multilayer GRUs. However, in minimalist frameworks like JAX's Flax, functionalities for stacking layers or integrating dropout are not provided out of the box, requiring developers to manually build upon built-in single-layer cells by programming the iterative layer logic and injecting dropout operations.

Concise Implementation of Deep RNNs

In deep, or stacked, Recurrent Neural Networks (RNNs), the hidden state is responsible for propagating sequential information throughout the multi-layered architecture. At any specific time step, the hidden state computed by a given layer is passed in two distinct directions: horizontally to the next time step within the identical layer, and vertically to the current time step of the subsequent layer. This two-dimensional information flow enables the network to build hierarchical representations over time.

Hidden State Propagation in Deep RNNs

Training deep Recurrent Neural Networks (RNNs) is a complex process that demands careful configuration to achieve proper convergence. Because the multi-layered sequential architecture exacerbates gradient-related issues, practitioners must ensure meticulous parameter initialization before training begins. Furthermore, successfully optimizing deep RNNs typically requires significant manual tuning and the application of specialized techniques, most notably learning rate adjustments and gradient clipping, to stabilize the learning process.

Training Requirements for Deep RNNs

Deep recurrent neural networks can be constructed using various types of recurrent units to process sequential data. The most common flavors of these deep architectures include standard vanilla RNNs, Long Short-Term Memory (LSTMs) networks, and Gated Recurrent Units (GRUs). These different varieties of recurrent models are conveniently accessible and can be instantiated as pre-built components through the high-level APIs of modern deep learning frameworks.

Flavors of Deep RNNs

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer. The initial layers of stacked networks can induce representations that serve as useful abstractions for further layers — representations that might prove difficult to induce in a single RNN. So it can outperform single-layer networks.

University of Michigan - Ann Arbor

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

Bidirectional RNNs

Stacked RNNs

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

NTMs(Neural Turing Machines) are neural network architectures that use external memory. These models are able to infer algorithmic tasks like copying, sorting, associative recall. According to the inventors of this architecture via NTMs they: "extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes"
These models are relatively new compared to LSTMs and RNNs. They were created in 2014. 

Neural Turing Machines (NTM)

https://arxiv.org/pdf/1410.5401.pdf?fbclid=IwAR1mft01ZW2dGNGyVczj4HR6E5H8xKpqTlniZeuI8JxGuSNzOdoNupq4j9M

Neural Turing Machines - Original Paper Reference

In the late 1990s, researchers Hochreiter and Schmidhuber proposed the Long Short-Term Memory (LSTM), which enables Recurrent Neural Networks (RNNs) to retain information over extended sequences rather than merely between consecutive time steps. Originally published in 1997, LSTMs gained significant recognition through victories in prediction competitions during the mid-2000s and became the dominant architecture for sequence learning from 2011 until the rise of Transformer models beginning in 2017. Even Transformers owe some of their key ideas to architectural design innovations first introduced by the LSTM. An LSTM-based RNN shares the same high-level architecture as a basic RNN (whether simple, bidirectional, or deep), but replaces standard activation functions with specialized LSTM cells.

Learn Before

Related

Learn After