RNNs are currently being applied to several applications, including:

 - Temporal analysis—time-series anomaly detection and time-series prediction
 - Computer vision—Image description, video tagging and video analysis
 - NLP—Sentiment analysis, speech recognition, language modeling, machine translation and text generation

Applications of RNN

The simplest form of RNN (“vanilla” RNN) is similar to a regular neural network, only it contains a loop that allows the model to carry forward results from previous neuron layers.

RNN Basic Structure

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

Both RNNs and HMMs make use of dynamic state. This means that these models depend both on the input system and current state. Additionally, RNNs have distributed state acquiring them with larger memory and computational capacity(which isn't true about HMMs). 

RNNs(Recurrent Neural Networks) vs HMMs (Hidden Markov Models)

Unlike feedforward neural networks:

1. Make use of sequence of inputs by using their memory
2. Cost function is applied at each time step unlike feedforward neural networks were cost is applied only to the final output. 

RNNs vs Feedforward Neural Networks

CNNs and RNNs are not mutually exclusive, as both can perform classification of image and text inputs, creating an opportunity to combine the two network types for increased effectiveness. This is especially true if the input to be classified is visually complex with added temporal characteristics that a CNN alone would be unable to process.

Typically, when these two network types are combined, sometimes referred to as a CRNN, inputs are first processed by CNN layers whose outputs are then fed to RNN layers. CNN Long Short-Term Memory (LSTM) architectures are particularly promising, as they facilitate analysis of inputs over longer periods than could be achieved with lower-level RNN architecture types.

Currently, these hybrid architectures are being explored for use in applications like video scene labeling, emotion detection or gesture recognition, video identification or gait recognition, and DNA sequence prediction.

Hybrid of Convolutional and Recurrent Neural Network

Why is an RNN (Recurrent Neural Network) used for machine translation, say translating English to French? (Check all that apply.)

In an RNN model, predictions are only based on knowledge/features/inputs from previous nodes. However, information from the rest of the sentence might be useful in the previous step as well.




Example: In the following examples, name entity recognition of Teddy can be different based on information that come after 'Teddy".




He said, "Teddy Roosevelt was a great president"
He said, "Teddy bear are on sale!"

RNN Problem

If we consider Tx the length of input and Ty the length of output:

1- Many-to-many ( Tx=Ty)

2- Many-to_many (Tx!=Ty)

3- Many-to-one

4-One_to_Many


Different types of RNN (in terms of input/output)

Gradients propagated many times tend to vanish or explode. Recurrent networks involve the composition of the same function many times which can result in nonlinear behavior. The figure shows the function after each time the transition function has been composed.



Long Term Dependencies Problem

"Conditioning" in the context of sequence to sequence learning in RNNs is the process of computing the probability of obtaining the output sequence conditioned on the input sequence, or p(y|x). The network is used to model this conditional probability mapping

We can extend a model to represent a distribution P(y|x) by using 3 common methods:

* As an extra input at each time step, or
* As the initial state h(0), or
* Both


Modeling Sequences Conditioned on Context with RNNs

As mentioned in the introduction to long-term dependencies, there are 3 ways to address long-term dependencies, and 1 of them is "Leakage units and other multi-timescale strategies", which will be introduced in this unit

This idea has led to numerous approaches, I will introduce three approaches here.

1 - Adding Skip Connections through Time

Adding direct connections from variables in the distant past to current variables is one way to get coarse time scales.
The gradient may vanish or exponentially about the time step. The algorithm introduces circular connections with d-delay to alleviate this problem. The rate of exponential decrease of the derivative is now related to τ/d correlation rather than τ. Since there is both a delay and a single-step connection, the gradient may still explode exponentially into t. This allows the learning algorithm to capture longer dependencies.

2- Leaky Units and a Spectrum of Diﬀerent Time Scales

The idea is similar to the sliding average model and shadow variables in TensorFlow.
The sliding average actually contains information over a long period of time.
In the long-term dependence problem, we can use the sliding average instead of the real parameters, making more long-term information to be considered when the parameters are iterated.

3- Removing connection

Unlike skiping connections, which add edges, the unit can self-select favorable dependencies. Whereas a removing connection directly removes the connection of length 1, forcing the connection to be longer.

Leaky Units and Other Strategies for Multiple Time Scales

A Convolutional Recurrent Neural Network (CRNN) is a combination of a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN). 
-CNN are specialized networks for processing data that use a specialized, linear mathematical operation, also known as a convolution. 
-RNN is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time adds additional layers of comprehension on top of the previous inputs. 
-The goal of a CRNN is to take advantage of CNNs for local feature extraction and RNNs for temporal summarization of the extracted features.


Convolutional Recurrent Neural Network (CRNN) 

For the parameters of all hidden states in a recurrent neural network, a pooling function can be applied to take their element-wise mean:

$\mathbf{h}_{\text {mean }}=\frac{1}{n} \sum_{i=1}^{n} \mathbf{h}_{i}$

Element-wise max values is also an option for the pooling function.

Pooling Layer in RNN

One reason for the inability of RNNs to carry forward critical information is that the hidden layers are being asked to perform two tasks simultaneously: 
- Provide information useful for the current decision
- Updating and carrying forward information required for future decisions

Inability of RNNs to Carry Forward Critical Information

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer, as shown in Fig. 9.10.

Stacked RNNs generally outperform single-layer networks. One reason for this
success seems to be that the network induces representations at differing levels of abstraction across layers. The optimal number of stacked RNNs is specific to each application and to each training set. However, as the number of stacks is increased the training costs rise quickly.

Stacked RNNs

A bidirectional RNN combines two independent bidirectional RNN
independent RNNs, one where the input is processed from the start to the end, and the other from the end to the start. We then concatenate the two representations computed by the networks into a single vector that captures both the left and right contexts of an input at each point in time. 

Fig. 9.11 illustrates such a bidirectional network that concatenates the outputs of the forward and backward pass. Other simple ways to combine the forward and backward contexts include element-wise addition or multiplication. The output at each step in time thus captures information to the left and to the right of the current input. In sequence labeling applications, these concatenated outputs can serve as the basis for a local labeling decision.

Bidirectional RNNs

Recurrent neural networks, or RNNs, are a family of neural networks for processing sequential data. A recurrent neural network is specialized for processing a sequence of inputs $x^{(1)}, . . . , x^{(τ)}$ and each time add additional layers of comprehension on top of the previous inputs. 


University of Michigan - Ann Arbor

San Diego State University

Deep learning algorithms can be implemented to solve real-world problems today.

Deep feedforward networks, also called multilayer perceptrons (MLPs), is the quintessential deep learning model that is behind nearly all modern practical applications of deep learning.

Scaling these models to large inputs such as high-resolution images or long temporal sequences requires specialization. 
 - Convolutional neural networks (CNN) can be used for processing large images.
 - Recurrent neural networks (RNN) can be used for processing temporal sequences. 


Deep Learning Algorithms

1. Models of Human Memory
2. Spaced Repetition
3. Leitner System
4. Supermemo System
5. Intelligent Tutoring System
6. Relation between Tutoring Systems and Student Learning
7. Reinforcement Learning
8. Trust Region Policy Optimization
9. Truncated Natural Policy Gradient
10. Recurrent Neural Networks

Related Theory (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

Sequence models are statistical supervised learning models that predict/classify based on sequence data.

Similar to other supervised statistical models, sequence models are in two types:
- Sequence prediction
- Sequence classification

Sequence Models

Goodfellow, I., Bengio, Y., & Courville, A. (2016). $\mathit{Deep \ Learning.}$ MIT Press. Retrieved from [www.deeplearningbook.org](https://www.deeplearningbook.org) 

Deep Learning

A helpful website that introduces neural networks:
https://missinglink.ai/guides/neural-network-concepts/

Neural Network Reference

Convolutional networks, also known as convolutional neural networks, or CNNs, are a specialized kind of neural network for processing data that has a known grid-like topology. Examples include time-series data, which can be thought of as a 1-D grid taking samples at regular time intervals, and image data, which can be thought of as a 2-D grid of pixels. 

The network employs a mathematical operation called convolution. Convolution is a specialized kind of linear operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their layers.

The CNN gets its power from the following concepts:
- Sparse connectivity
- Parameter sharing

Convolutional Neural Network (CNN)

Recurrent Neural Network (RNN)

**Generative Models**  As might be clear from their name , generative models are able to generate new contents by learning features of certain datasets. As for more formal definition, generative models specify in terms of a probabilistic model describing how it thinks the dataset was produced.  Using sampling from this probability distribution, we can create new data in the style of the old.
 
The joint probability distribution P(X,Y) is learned from the data, and then divided by P(X) to obtain the conditional probability distribution P(Y|X), which is used as a predictive model. It is called a generative model because the model represents the relationship between a given input X and an output Y. Common generative models are: Naive Bayes, Hidden Markov Model, Gaussian Mixture Model, Document Topic Generation Model (LDA), and restricted Boltzmann machine.

Generative Models

There are functions you can compute with a small layer that shallower networks would need a lot more hidden units in order to compute. 

Circuit Theory

https://www.simplilearn.com/deep-learning-algorithms-article

More about deep learning algorithms 

Data inputted into deep learning algorithms should be split into a distribution in which the majority of data (well over 70%+) is fed into the training set. This differs from traditional machine learning algorithms which had ~70% of the data given to the training set. This difference comes from the fact that there is now much more data available and deep learning algorithms improve with more data, whereas traditional machine learning algorithms improvement levels out after a certain amount of data is fed in.

For example, in a dataset of 1 million examples, you mighht decide that 10k examples are enough to evaluate which algorithm is better - 98% train, 1% dev, 1% test.

Deep learning train/dev/test split 

Deep feedforward networks, also called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models.

It was the first and simplest type of artificial neural network devised. In this network, the information moves in only one direction, forward, from the input nodes, through the hidden nodes (if any) and to the output nodes. There are no cycles or loops in the network.

Deep Feedforward Networks (MLP = Multi-Layer Perceptrons)

Instead of manually building up deep network architectures, many python libraries are available online to help users to easily implement deep learning models.

- Trax (built on TensorFlow)
- Tensor2Tensor (built on TensorFlow)
- Pytorch
- Fast.ai ( built on top of PyTorch)
- Keras (built to be a higher level framework to interface with other frameworks like TensorFlow)
- Vowpal Wabbit 
- Theano
- Caffee 

Deep Learning Python libraries (frameworks)

What can convolutional neural networks be used for?

A generative adversarial network (GAN) is a class of machine learning framework where two neural networks, a generative model and a discriminant model, contest against each other in a "game" (in the form of a zero-sum game, where one agent's gain is another agent's loss). Given a training set, this technique learns to generate new data with the same statistics as the training set. 

The core idea of a GAN is based on the "indirect" training through the discriminator, which itself is also being updated dynamically. This basically means that the generator is not trained to minimize the distance to a specific image, but rather to fool the discriminator and the discriminator is rewarded every time it correctly deduces if a sample is from the model distribution or the data distribution. This enables the model to learn in an unsupervised manner.

Generative adversarial network(GAN)

Deep Learning models require incredible computational resources to successfully train a model. As a result, optimizing the training process is a key step in developing an algorithm. One approach involves finding the parameters of a neural network that minimize its loss function. This is accomplished via gradient descent, a popular technique for minimizing loss function error. 

Optimization for Training Deep Models

Whereas deep learning algorithms were once always implemented on a single computer's central processing unit (CPU), research has shown that families of CPUs or even a single graphics processing unit (GPU) is much more effective. Some systems employ even greater parallelization to achieve the fastest training times with distributed architectures.

Implementations of Deep Learning

Monte Carlo algorithms return results with variability. The variability can typically be reduced by allocating more resources. The benefit of Monte Carlo algorithms is that they can provide an approximated estimation with limited source while maintaining a certain degree of accuracy. In Machine Learning, many problems are too complicated to have precise answers. Monte Carlo approximations are therefore a good approach to these problems.

Monte Carlo Methods

Deep learning frameworks provide specialized software tools that streamline the construction, training, and deployment of neural networks. The availability of these efficient frameworks has made the design and implementation of whole system optimization significantly easier, which is a critical component in achieving high performance for complex deep learning pipelines.

Deep Learning Frameworks

An adversarial example is an input to a machine learning model that has been intentionally modified with small, often imperceptible perturbations designed to cause the model to output an incorrect classification. For instance, adding specific, calculated noise to an image of a panda can cause a deep neural network to misclassify it as a gibbon, even though the image remains visually identical to the original for a human observer.

Adversarial Example

Here the authors describe three previous probabilistic methods of the human memory:

**Exponential Forgetting Curve**  - one of the oldest human memory models.
The authors use parameterization where user either forgets information completely or retains it and the probability of the recall can be expressed with this function:  

"where θ ∈ R+ is the item difficulty, D ∈ R+ is the time elapsed since the item was last reviewed by the student, and S ∈ R+ is the student’s memory strength for the item." S is set to the amount of trials. 

                                       $$Z ~ Bernoulli(exp(-\theta \frac {D}{S}))$$
**Half-Life Regression** - here unlike **exponential forgetting curve** we don't have the parameter for the difficulty here, instead here we have ~x ∈ X which contains information about student's study history and model parameters θ ∈ Θ . This is how formula looks like:
                                 $$ S = exp(\overrightarrow { \rm \theta} \overrightarrow {\rm x})$$

In order to encode the number of attempts, correct/incorrect answers and the identity  the authors set $X = \N ^{3} \times {0, 1} ^{n}$. By dropping difficulty of the item we are not losing any information as the difficulty "is absorbed into the memory strength via the coefficients of the item identity indicator features."

**Generalized Power Law** (paper describes this part precisely, couldn't be broken down into smaller parts) here we have different likelihood for the recall:


On the last formula from the picture we have a and d  which are parameters for the student ability and the item difficulty.


Background
(Accelerating Human Learning With Deep Reinforcement Learning)

Reviewing or practicing over time can have better results than reviewing in short periods of time.

Spaced Repetition

The method used in flashcards for the spaced repetition. 

**The Goal** focus more on those items with which students have more difficulties to recall and spend less time on the ones that student recalls.

Leitner System

This is the algorithm for the supermemo (**SP2**) which was described in the paper:

1. Split knowledge

2. E-Factor = 2.5, for all items

3. Repeat items using the following intervals:
    I(1):=1
    I(2):=6
    for n>2: I(n):=I(n-1)*EF
    where:
    I(n) - inter-repetition interval after the n-th repetition (in days),
    EF - E-Factor (easiness of memorizing and retaining a given item in memory)

4. There is grade scale 0-5:
     5 - PERFECT
     4 - Correct + hesitation
     3 - Correct + took more time
     2 - Incorrect + correct easy to recall
     1 - Incorrect + correct  remembered
     0 - Blackout

5. After this we change E-Factors using the formula:
     EF_new:=EF+(0.1-(5-q)*(0.08+(5-q)*0.02))
     q - 0-5 range; EF_new - new E-Factor, if EF < 1.3 $\rightarrow$ EF = 1.3

6. If q < 3 then repeat 1-5 without changing EF.

7. After on a specific day the session is finished, repeat the procedure until at least q=4 is achieved for all the items.

Supermemo System

In reinforcement learning, the machine goes through trial and error processes where it is rewarded/penalized for the actions it performs. The machine's goal is to maximize the total reward by leveraging previous attempts to make the next decision. The model starts from completely random trials and eventually leads to something with very sophisticated tactics and skills. Compared to supervised/unsupervised learning, reinforcement learning is much more advanced. A reinforcement model will continuously learn, unlike supervised/unsupervised models which all have an endpoint after the training and test data phases.

Reinforcement Learning

In the 21st century MOOC (Massive Open Online Courses) have proliferated like Coursera, Udemy, Edx and several applications like Duolingo, Quizzlet became quite popular.  The researchers have a keen interest to acquire this systems with more intelligence, so that learning process could be overseen. 

As it was described in the paper Computer-Assisted Instruction can increase students performance by 0.3 std while human tutors by 2. 

Intelligent Tutoring Systems combine AI, cognitive science and educational theory which makes researches in this field arduous.  

In the earlier studies 3 main modules have been defined in ITS:

1. **The expert Knowledge module** - this module can be thought as a source of the knowledge. The purpose of this module is to generate questions, answers and the solutions for the particular problem.

2. **The student model module**  - the student skills and knowledge which varies constantly. It is very important for the **tutor module** to know about students current level and skills.

3. **The tutor module** - the teaching strategy. This module has to decide which lesson to present to the student.

Recently, fourth module was added **The user interface module**:



Intelligent Tutoring Systems (Using deep reinforcement learning for personalizing review sessions on e-learning platforms with spaced repetition)

It is extremely important for the tutoring systems to realize the student's current knowledge level and predict student's ability to answer/solve a particular task or question correctly and modify student's knowledge level accordingly. The tutoring system has to take care of providing questions. System should take lots of factors into account like: 

1. Duration (to learn material)
2. Manner (to learn material)

Relation between Tutoring Systems and Student learning

TRPO is an optimization algorithm in reinforcement learning which uses gradient descent. TRPO builds an algorithm that is stable and guarantees monotonic improvement. "This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks"

TRPO has  better performance than the vanilla policy gradients as the length of the step size is easily defined. Additionally, it takes advantage of using old policies sampled distributions for optimizing new ones.



Trust Region Policy Optimization

**Truncated Natural Policy Gradient** (TNPG) is the algorithm for computing gradient direction. It only needs to calculate $I(\theta)v$ where:

1. $I (\theta)$ is Fisher Information Matrix
2. v is arbitrary vector.

TNPG improves natural policy gradient but it has a high computational cost. Usually, they are efficient when the space of the parameters is high dimensional. 

Truncated Natural Policy Gradient

https://machinelearningmastery.com/sequence-prediction/

Predictions with Sequences

Sequence prediction is used to predict the next value of a given input sequence. One example is product recommendations, which predicts a customer's next purchase given a sequence of their past purchases. Another example is stock market prediction.

Sequence Prediction Models

Sequence classification predicts a class label for a given input sequence. Sentiment analysis is one example, which takes a sequence of text and predicts the label for the sentiment (positive or negative). 

Sequence Classification Models

You have trained word embeddings using a text dataset of $$m_1$$ words. You are considering using these word embeddings for a language task, for which you have a separate labeled dataset of $$m_2$$ words. Keeping in mind that using word embeddings is a form of transfer learning, under which of these circumstance would you expect the word embeddings to be helpful?

Sequence Model Question #1

In the word2vec algorithm, you estimate $$P\left(t|c\right)$$, where t is the target word and c is a context word. How are t and c chosen from the training set?

Sequence Model Question #2

Suppose you learn a word embedding for a vocabulary of 10000 words. Then the embedding vectors should be 10000 dimensional, so as to capture the full range of variation and meaning in those words.

Sequence Moel Question #4

In beam search, if you increase the beam width B, which of the following would you expect to be true? Check all that apply.

Sequence Model Question #3

Tokenization is the process of converting a sequence of text into smaller units, known as tokens. It is a foundational step in Natural Language Processing, and there are numerous different methods and strategies for how a text can be tokenized.

Tokenization

In sequence modeling architectures, standard notation is used to represent the inputs and outputs. A source sequence is typically denoted as $$x_1...x_m$$, where $$m$$ represents the length of the source sequence. Its corresponding sequence of embedding vectors is denoted as $$\mathbf{e}_{1}^{x}...\mathbf{e}_{m}^{x}$$. Similarly, the target sequence that the model aims to generate is denoted as $$y_1...y_n$$, with length $$n$$, and its corresponding embedding sequence is represented as $$\mathbf{e}_{1}^{y}...\mathbf{e}_{n}^{y}$$.

Learn Before

Related

Learn After