The long short-term memory (LSTM) network is an extension to RNNs, in addressing problems such as the vanishing gradient problem and the inability of RNNs to carry forward critical information.

LSTMs divide the context management problem into two sub-problems: 
- Removing information no longer needed from the context
- Adding information likely to be needed for later decision making

Long Short-Term Memory (LSTM) Network

One reason for the inability of RNNs to carry forward critical information is that the hidden layers are being asked to perform two tasks simultaneously: 
- Provide information useful for the current decision
- Updating and carrying forward information required for future decisions

University of Michigan - Ann Arbor

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


Recurrent Neural Network (RNN)

An on-going but a helpful book resource about NLP
https://web.stanford.edu/~jurafsky/slp3/

Speech and Language Processing (3rd ed. draft) 

RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

Both RNNs and HMMs make use of dynamic state. This means that these models depend both on the input system and current state. Additionally, RNNs have distributed state acquiring them with larger memory and computational capacity(which isn't true about HMMs). 

RNNs(Recurrent Neural Networks) vs HMMs (Hidden Markov Models)

Unlike feedforward neural networks:

1. Make use of sequence of inputs by using their memory
2. Cost function is applied at each time step unlike feedforward neural networks were cost is applied only to the final output. 

RNNs vs Feedforward Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

Why is an RNN (Recurrent Neural Network) used for machine translation, say translating English to French? (Check all that apply.)

In an RNN model, predictions are only based on knowledge/features/inputs from previous nodes. However, information from the rest of the sentence might be useful in the previous step as well.




Example: In the following examples, name entity recognition of Teddy can be different based on information that come after 'Teddy".




He said, "Teddy Roosevelt was a great president"
He said, "Teddy bear are on sale!"

RNN Problem

If we consider Tx the length of input and Ty the length of output:

1- Many-to-many ( Tx=Ty)

2- Many-to_many (Tx!=Ty)

3- Many-to-one

4-One_to_Many


Different types of RNN (in terms of input/output)

Gradients propagated many times tend to vanish or explode. Recurrent networks involve the composition of the same function many times which can result in nonlinear behavior. The figure shows the function after each time the transition function has been composed.



Long Term Dependencies Problem

"Conditioning" in the context of sequence to sequence learning in RNNs is the process of computing the probability of obtaining the output sequence conditioned on the input sequence, or p(y|x). The network is used to model this conditional probability mapping

We can extend a model to represent a distribution P(y|x) by using 3 common methods:

* As an extra input at each time step, or
* As the initial state h(0), or
* Both


Modeling Sequences Conditioned on Context with RNNs

As mentioned in the introduction to long-term dependencies, there are 3 ways to address long-term dependencies, and 1 of them is "Leakage units and other multi-timescale strategies", which will be introduced in this unit

This idea has led to numerous approaches, I will introduce three approaches here.

1 - Adding Skip Connections through Time

Adding direct connections from variables in the distant past to current variables is one way to get coarse time scales.
The gradient may vanish or exponentially about the time step. The algorithm introduces circular connections with d-delay to alleviate this problem. The rate of exponential decrease of the derivative is now related to τ/d correlation rather than τ. Since there is both a delay and a single-step connection, the gradient may still explode exponentially into t. This allows the learning algorithm to capture longer dependencies.

2- Leaky Units and a Spectrum of Diﬀerent Time Scales

The idea is similar to the sliding average model and shadow variables in TensorFlow.
The sliding average actually contains information over a long period of time.
In the long-term dependence problem, we can use the sliding average instead of the real parameters, making more long-term information to be considered when the parameters are iterated.

3- Removing connection

Unlike skiping connections, which add edges, the unit can self-select favorable dependencies. Whereas a removing connection directly removes the connection of length 1, forcing the connection to be longer.

Leaky Units and Other Strategies for Multiple Time Scales

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

For the parameters of all hidden states in a recurrent neural network, a pooling function can be applied to take their element-wise mean:

$\mathbf{h}_{\text {mean }}=\frac{1}{n} \sum_{i=1}^{n} \mathbf{h}_{i}$

Element-wise max values is also an option for the pooling function.

Pooling Layer in RNN

Inability of RNNs to Carry Forward Critical Information

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer, as shown in Fig. 9.10.

Stacked RNNs generally outperform single-layer networks. One reason for this
success seems to be that the network induces representations at differing levels of abstraction across layers. The optimal number of stacked RNNs is specific to each application and to each training set. However, as the number of stacks is increased the training costs rise quickly.

Stacked RNNs

A bidirectional RNN combines two independent bidirectional RNN
independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start. We then concatenate the two representations computed by the networks into a single vector that captures both the left and right contexts of an input at each point in time. 

Fig. 9.11 illustrates such a bidirectional network that concatenates the outputs of the forward and backward pass. Other simple ways to combine the forward and backward contexts include element-wise addition or multiplication. The output at each step in time thus captures information to the left and to the right of the current input. In sequence labeling applications, these concatenated outputs can serve as the basis for a local labeling decision.

Learn Before

Related

Learn After