1. We can choose weighting by content-based system without any modification
2. We can choose weighting by content-based system and apply shifting 
3. "Weighting from the previous time step can be rotated without any input from the content-based addressing system"

Advantages of Using both Content and Location Based Addressing (NTM Architecture) 

In the Neural Turing Machine (NTM) architecture, content-based addressing focuses on memory locations based on the similarity between values emitted by the controller and the contents of memory. For both read and write operations, the controller first emits a key vector of length $$M$$, denoted as $$k_t$$. This key vector is compared to each memory vector, $$M_t(i)$$.

This method produces normalized weights, $$w_t^c(i)$$, using a positive key strength, $$\beta_t$$:
$$w_t^c(i) = \frac {\exp(\beta_t K[k_t, M_t(i)])} {\sum_j \exp(\beta_t K[k_t, M_t(j)])}$$

The similarity is measured using cosine similarity:
$$K[u, v] = \frac {u \cdot v} {\|u\| \|v\|}$$

Content-Based Addressing (NTM Architecture)

When the content-based method is not well-suited for a problem, location-based addressing is used, which purely focuses on memory location using rotational shifting and weighting. Before rotational shifting is performed, both read and write heads produce a scalar interpolation gate $$g_t$$ (between 0 and 1). This value acts as a blend between $$w_{t-1}$$ (the weight produced in the previous time step by the read/write head) and $$w_t^c$$ (generated by the content-based system) to return a gated weight, $$w_t^g$$:

$$w_t^g \leftarrow g_t w_t^c + (1 - g_t) w_{t-1}$$

Depending on the value of $$g_t$$, we might completely ignore weights produced by the content-based system or by the head in the previous time step. More precisely, if $$g_t$$ is equal to 0, we ignore the content-based system, and if it is 1, we ignore the previous head weight. After this procedure, shift weighting $$s_t$$ is applied. The simplest way to define the shift weighting is to use a softmax distribution. The rotation applied to the gated weight is written in the following formula:

$$w_t^{\sim} \leftarrow \sum_{j=i}^{N-1} w_t^g(j) s_t(i-j)$$

If weights aren't sharp, this convolution procedure can lead to leakage or dispersion. In order to solve this problem, each head produces an additional scalar $$\gamma_t \ge 1$$ which makes sure to sharpen those weights:

$$w_t(i) \leftarrow \frac{w_t^{\sim}(i)^{\gamma_t}}{\sum_j w_t^{\sim}(j)^{\gamma_t}}$$

Location-Based Addressing (NTM Architecture)

The weights are produced using two addressing mechanisms that are:
1. Content-based 
2. Location-based 

San Diego State University

NTMs have two basic components, namely, controller and memory bank. Controller uses input and output vectors to interact with outside environment. This is similar to what we have in typical neural network architectures. Additionally, unlike other neural network architectures, it also interacts with the memory bank using read and write operations. 
NTMs can be used with gradient descent and this process would be quite straightforward as each component is differentiable.



NTM Architecture

Addressing Mechanisms (NTM Architecture)

The most important architectural choice of the NTMs is the type of the neural network that would be used as a controller. More precisely, we have to decide whether we want to use feedforward or recurrent network. 
"If one compares the controller to the central processing unit in a digital computer (albeit with adaptive rather than predefined instructions) and the memory matrix to RAM, then the hidden activations of the recurrent controller are akin to the registers in the processor"
It is interesting that if we choose feedforward networks as a controller we can mimic RNNs by reading and writing at the same location at every time step.

Controller Network

In Neural Turing Machines (NTMs), components are differentiable because the read and write operations are 'blurry.' Instead of addressing a single memory element like a standard Turing machine or digital computer, NTM operations interact to varying degrees with all elements in the memory. An attentional focus mechanism determines this degree of blurriness, limiting input and output interactions to relatively small parts of the memory while neglecting the rest.

Differentiability of NTM Components

In the Neural Turing Machine (NTM) architecture, a read operation at time $$t$$ uses an $$N \times M$$ memory matrix, denoted as $$M_t$$, and a vector of weightings, $$w_t$$, over the $$N$$ memory locations emitted by a read head. The weightings are normalized such that:

$$\sum_i w_t(i) = 1, \quad 0 \le w_t(i) \le 1 \quad \forall i$$

The read vector, $$r_t$$, returned by the read operation is defined as the convex combination of the memory row vectors:

$$r_t \leftarrow \sum_i w_t(i)M_t(i)$$

Reading (NTM Architecture)

In the Neural Turing Machine (NTM) architecture, each write operation is decomposed into two parts: an **erase** phase and an **add** phase. Given an erase vector $$e_t$$ (whose $$M$$ elements lie in the range $$(0, 1)$$) and a weight vector $$w_t$$ emitted by the write head, the previous memory vectors $$M_{t-1}(i)$$ are modified as follows: $$M_{t}^{\sim}(i) \leftarrow M_{t-1}(i)[1 - w_t(i)e_t]$$ If both the weight and erase values at a location are equal to one, the memory is reset to 0; otherwise, the memory is partially erased or remains unchanged. After the erase step, an add vector $$a_t$$ is applied to update the memory: $$M_t(i) \leftarrow M_{t}^{\sim}(i) + w_t(i)a_t$$

Learn Before

Related

Learn After