Even though, computer programs heavily rely on logical flow control, elementary operations and external memory, the modern machine learning almost never uses two of these fundamental mechanisms: logical flow control and external memory.

Motivation for Creating Neural Turing Machines

NTMs have two basic components, namely, controller and memory bank. Controller uses input and output vectors to interact with outside environment. This is similar to what we have in typical neural network architectures. Additionally, unlike other neural network architectures, it also interacts with the memory bank using read and write operations. 
NTMs can be used with gradient descent and this process would be quite straightforward as each component is differentiable.



NTM Architecture

"The copy task tests whether NTM can store and recall a long sequence of arbitrary information. The network is presented with an input sequence of random binary vectors followed by a delimiter flag. Storage and access of information over long time periods has  always been problematic for RNNs and other dynamic architectures. We were particularly interested to see if an NTM is able to bridge longer time delays than LSTM"

Copy with NTMs

Repeat copy is similar to the copy, additionally here we also want to get an output at the specified number of times. The authors wanted to test if NTMs would be able to learn trivial nested functions. 


Repeat Copy with NTMs

"We test the NTM’s capability for learning an instance of this more interesting class by constructing a list of items so that querying with
one of the items demands that the network return the subsequent item. More specifically, we define an item as a sequence of binary vectors that is bounded on the left and right by delimiter symbols. After several items have been propagated to the network, we query by showing a random item, and we ask the network to produce the next item. In our experiments, each item consisted of three six-bit binary vectors"

Associative Recall with NTMs

The authors wanted to test how NTMs would perform as re-writable tables, like N-Grams. In this case all possible 6-Gram distribution over binary sequences where used. The network, firstly, observes data one bit at a time and then it has to predict next bit. 

N-Grams with NTMs

As it clear from the title, the authors tested how NTMs would perform when it comes to  simple sorting algorithm. The priority is chosen from range [-1, 1]. Input contained binary vectors with their priorities and target contained sorted input. based on their priorities.
 

Priority Sort with NTMs

In terms of training time NTM is not better than LSTM. Furthermore, there are lot's of things going inside NTM which requires more iterations than we would need for LSTM.
In terms of performance, NTM is more likely to beat the LSTM.  Theoretically, NTMs can have an infinite memory across timestamps, thus they are  good when we have large data (**this might not be true in practice**). However, if we have smaller data LSTMs can be a better choice. Therefore depending on the task and the amount of input NTM can be better than LSTM.

https://www.quora.com/Is-the-Neural-Turing-machine-more-efficient-than-the-LSTM?fbclid=IwAR1P-OEKHbp3wCW9FUKzflCSlsJb8S5gcKmIa5wtYezT_dsuC1Ov-fH0sVQ

NTMs vs LSTMs

1. Ability to learn algorithms
2. Ability to generalize
3. Has a good performance even if we have a huge data

Advantages (Neural Turing Machines)

1. Time complexity
2. Difficult to implement
3. Doesn't benefit much from GPU
4. Architecture dependent

Disadvantages (Neural Turing Machines)

NTMs(Neural Turing Machines) are neural network architectures that use external memory. These models are able to infer algorithmic tasks like copying, sorting, associative recall. According to the inventors of this architecture via NTMs they: "extend the capabilities of neural networks by coupling them to external memory resources, which they can interact with by attentional processes"
These models are relatively new compared to LSTMs and RNNs. They were created in 2014. 

San Diego State University

RNNs come in many variants, including:
 - Bidirectional RNNs
 - Memory Units (LSTMs) RNNs
 - Fully recurrent 
 - Elman/Jordan networks, or Simple Recurrent Networks (SRN)
 - Recursive neural network
 - Hopfield
 - Echo State Network (ESN)
 - Stacked RNNs
 - Hierarchical
 - Neural Turing machines (NTMs)
 - Differentiable Neural Computer (DNC) 
 - Recurrent Multi-Layer Perceptron (RMLP)
 - Independent RNN (IndRNN)
 - Neural history compressor 
 - Second order RNNs 
 - Gated recurrent unit (GRU)
 - Continuous-Time Recurrent Neural Network (CTRNN)
 - Multiple Timescales Recurrent Neural Network (MTRNN) 
 - Neural Network Pushdown Automata (NNPDA)

RNN Extensions and Types

https://arxiv.org/pdf/1410.5401.pdf?fbclid=IwAR1mft01ZW2dGNGyVczj4HR6E5H8xKpqTlniZeuI8JxGuSNzOdoNupq4j9M

Neural Turing Machines - Original Paper Reference

A bidirectional RNN assumes that the correct output not only depends on the previous inputs in the time series but also on future inputs. For example, in translation models, it is often necessary to use a word from the end of the source sentence in order to predict a word early in the target sentence. 

To make this possible, two RNNs are stacked on top of each other, one going from beginning to end and the other from end to beginning, and the output is computed based on hidden states of both networks.

Bidirectional RNNs

Stacked RNNs consist of multiple networks where the output of one layer serves as the input to a subsequent layer. The initial layers of stacked networks can induce representations that serve as useful abstractions for further layers — representations that might prove difficult to induce in a single RNN. So it can outperform single-layer networks.

Stacked RNNs

Gated Recurrent Units (GRUs) resolve the vanishing gradients problem in simple RNNs and also ease the burden of introducing a considerable number of additional parameters in the LSTMs, by dispensing with the use of a separate context vector, and by reducing the number of gates to 2 — a reset/relevance gate and an update gate.
The purpose of the reset gate is to decide which aspects of the previous hidden state are relevant to the current context and what can be ignored. It computes an intermediate representation for the new hidden state at the current time.
The purpose of the update gate is to determine which aspects of this new state will be used directly in the new hidden state and which aspects of the previous state need to be preserved for future use.

Gated recurrent unit (GRU)

Neural Turing Machines (NTM)

In the late 1990s, German researchers Hochreiter and Schmidhuber proposed the concept of Long Short-Term Memory (LSTM), which helps Recurrent Neural Networks (RNNs) retain information over extended sequences, rather than just between consecutive time steps. For example, a character’s name introduced at the beginning of a paragraph may be necessary to understand the context at the end. An LSTM-based RNN shares the same high-level architecture as a basic RNN (whether simple, bidirectional, or deep), but replaces standard activation functions with specialized LSTM cells.

Learn Before

Related

Learn After