A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

Bidirectional RNNs

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer. The initial layers of stacked networks can induce representations that serve as useful abstractions for further layers — representations that might prove difficult to induce in a single RNN. So it can outperform single-layer networks.

Stacked RNNs

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

NTMs(Neural Turing Machines) are neural network architectures that use external memory. These models are able to infer algorithmic tasks like copying, sorting, associative recall. According to the inventors of this architecture via NTMs they: "extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes"
These models are relatively new compared to LSTMs and RNNs. They were created in 2014. 

Neural Turing Machines (NTM)

https://arxiv.org/pdf/1410.5401.pdf?fbclid=IwAR1mft01ZW2dGNGyVczj4HR6E5H8xKpqTlniZeuI8JxGuSNzOdoNupq4j9M

Neural Turing Machines - Original Paper Reference

In the late 1990s, German researchers Hochreiter and Schmidhuber proposed the concept of Long Short-Term Memory (LSTM), which helps Recurrent Neural Networks (RNNs) retain information over extended sequences, rather than just between consecutive time steps. For example, a character’s name introduced at the beginning of a paragraph may be necessary to understand the context at the end. An LSTM-based RNN shares the same high-level architecture as a basic RNN (whether simple, bidirectional, or deep), but replaces standard activation functions with specialized LSTM cells.

LSTM-Based RNN Architecture

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

University of Michigan - Ann Arbor

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

RNN Extensions and Types

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

Both RNNs and HMMs make use of dynamic state. This means that these models depend both on the input system and current state. Additionally, RNNs have distributed state acquiring them with larger memory and computational capacity(which isn't true about HMMs). 

RNNs(Recurrent Neural Networks) vs HMMs (Hidden Markov Models)

Unlike feedforward neural networks:

1. Make use of sequence of inputs by using their memory
2. Cost function is applied at each time step unlike feedforward neural networks were cost is applied only to the final output. 

RNNs vs Feedforward Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

Why is an RNN (Recurrent Neural Network) used for machine translation, say translating English to French? (Check all that apply.)

In an RNN model, predictions are only based on knowledge/features/inputs from previous nodes. However, information from the rest of the sentence might be useful in the previous step as well.




Example: In the following examples, name entity recognition of Teddy can be different based on information that come after 'Teddy".




He said, "Teddy Roosevelt was a great president"
He said, "Teddy bear are on sale!"

RNN Problem

If we consider Tx the length of input and Ty the length of output:

1- Many-to-many ( Tx=Ty)

2- Many-to_many (Tx!=Ty)

3- Many-to-one

4-One_to_Many


Different types of RNN (in terms of input/output)

Gradients propagated many times tend to vanish or explode. Recurrent networks involve the composition of the same function many times which can result in nonlinear behavior. The figure shows the function after each time the transition function has been composed.



Long Term Dependencies Problem

"Conditioning" in the context of sequence to sequence learning in RNNs is the process of computing the probability of obtaining the output sequence conditioned on the input sequence, or p(y|x). The network is used to model this conditional probability mapping

We can extend a model to represent a distribution P(y|x) by using 3 common methods:

* As an extra input at each time step, or
* As the initial state h(0), or
* Both


Modeling Sequences Conditioned on Context with RNNs

As mentioned in the introduction to long-term dependencies, there are 3 ways to address long-term dependencies, and 1 of them is "Leakage units and other multi-timescale strategies", which will be introduced in this unit

This idea has led to numerous approaches, I will introduce three approaches here.

1 - Adding Skip Connections through Time

Adding direct connections from variables in the distant past to current variables is one way to get coarse time scales.
The gradient may vanish or exponentially about the time step. The algorithm introduces circular connections with d-delay to alleviate this problem. The rate of exponential decrease of the derivative is now related to τ/d correlation rather than τ. Since there is both a delay and a single-step connection, the gradient may still explode exponentially into t. This allows the learning algorithm to capture longer dependencies.

2- Leaky Units and a Spectrum of Diﬀerent Time Scales

The idea is similar to the sliding average model and shadow variables in TensorFlow.
The sliding average actually contains information over a long period of time.
In the long-term dependence problem, we can use the sliding average instead of the real parameters, making more long-term information to be considered when the parameters are iterated.

3- Removing connection

Unlike skiping connections, which add edges, the unit can self-select favorable dependencies. Whereas a removing connection directly removes the connection of length 1, forcing the connection to be longer.

Leaky Units and Other Strategies for Multiple Time Scales

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

For the parameters of all hidden states in a recurrent neural network, a pooling function can be applied to take their element-wise mean:

$\mathbf{h}_{\text {mean }}=\frac{1}{n} \sum_{i=1}^{n} \mathbf{h}_{i}$

Element-wise max values is also an option for the pooling function.

Pooling Layer in RNN

One reason for the inability of RNNs to carry forward critical information is that the hidden layers are being asked to perform two tasks simultaneously: 
- Provide information useful for the current decision
- Updating and carrying forward information required for future decisions

Inability of RNNs to Carry Forward Critical Information

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer, as shown in Fig. 9.10.

Stacked RNNs generally outperform single-layer networks. One reason for this
success seems to be that the network induces representations at differing levels of abstraction across layers. The optimal number of stacked RNNs is specific to each application and to each training set. However, as the number of stacks is increased the training costs rise quickly.

A bidirectional RNN combines two independent bidirectional RNN
independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start. We then concatenate the two representations computed by the networks into a single vector that captures both the left and right contexts of an input at each point in time. 

Fig. 9.11 illustrates such a bidirectional network that concatenates the outputs of the forward and backward pass. Other simple ways to combine the forward and backward contexts include element-wise addition or multiplication. The output at each step in time thus captures information to the left and to the right of the current input. In sequence labeling applications, these concatenated outputs can serve as the basis for a local labeling decision.

Learn Before

Related

Learn After