Here we focus on location based on the similarity between the values that were emitted by the controller.
Here both read/write firstly focuses on a length M key vector $K_t$ and we compare them to $M_t(i)$ vectors. This method produces normalized weights:
$$w_t^c(i) = \frac {exp(\beta_t K[k_t, M_t(i)])} {\sum_j \beta_t K[k_t, M_t(j)]}$$
The similarity is measured using cosine similarity:
$$k[u, v] = \frac {u v} {||u||  ||v||} $$


Content-based (NTM Architecture)

In case, content-based method isn't well-suited for the problem, location-based addressing is also used which purely focuses on the location.

Location-based addressing uses rotational shifting and weighting.
Before rotational shifting will be performed, both read/write heads produce scalar interpolation gate $g_t$ (between 0 and 1). This value acts like a blend between the $w_{t-1}$ (weight produced in the previous time step by read/write head)  and the $w_t^c$ generated by the content-based system and as the result gated weight
$w_t^g$ is returned:
$$w_t^g \leftarrow g_t w_t^c + (1 - g_t) w_{t-1}$$

Depending on the value of g we might completely ignore weights produced by content-based system or by head in the previous timestamp. More precisely, if g is equal to we ignore content-based system and if it is 1 we ignore head.

After this procedure, shift weighting $s_t$ is applied. The simplest way would be to use softmax for defining shift weighting. Rotation that is applied to our gated weight is written in the following formula:
$$w_t^{\sim} \leftarrow \sum_{j=i}^{N-1} w_t^g (j) s_t(i-j)$$ 
If weights aren't sharp this convolution procedure can lead to leakage or dispersion. In order to solve this problem, each head produces additional scalar $\gamma_t \ge 1$ which makes sure to sharpen those weights:
$$w_t(i)  \leftarrow \frac {w_t^{\sim} (i)^{\gamma_t}}{\sum_jw_t^{\sim}(j)^{\gamma_t}}$$

Location-based (NTM Architecture)

1. We can choose weighting by content-based system without any modification
2. We can choose weighting by content-based system and apply shifting 
3. "Weighting from the previous time step can be rotated without any input from the content-based addressing system"

Advantages of Using both Content and Location Based Addressing (NTM Architecture) 

The weights are produced using two addressing mechanisms that are:
1. Content-based 
2. Location-based 

San Diego State University

NTMs have two basic components, namely, controller and memory bank. Controller uses input and output vectors to interact with outside environment. This is similar to what we have in typical neural network architectures. Additionally, unlike other neural network architectures, it also interacts with the memory bank using read and write operations. 
NTMs can be used with gradient descent and this process would be quite straightforward as each component is differentiable.



NTM Architecture

Suppose that we have $M_t$ contents from NxM and $w_t$ is weightings over the N locations which where emitted by read head at time t. All weightings are normalized and we have these properties:
$$\sum_i w_t(i) = 1, \quad 0 \le w_t(i) \le 1  \quad \forall i$$

The $r_t$ that is returned by the read is defined as followes:
$$ r_t \leftarrow \sum_i w_t(i)M_t(i) $$

Reading (NTM Architecture)

Each write is decomposed into two parts: **erase** and **add**.

We have erase vector $e_t$  (its M elements lie in range (0, 1))and $w_t$ which is emitted by write head, memory vectors $M_{t-1}(i)$ are changed as it is written in the given formula:
$$M_{t}^{\sim}(i) \leftarrow M_{t-1}(i)[1 - w_t(i)e_t]$$
As it is clear from the formula if at the location both weight and erase are equal to one than we reset them to 0. In  other cases we don't change the memory. 

After erase we also have add vector $a_t$ that is added to our memory:
$$M_t(i) \leftarrow M_{t}^{\sim} + w_t(i)a_t$$


Writing (NTM Architecture)

Addressing Mechanisms (NTM Architecture)

The most important architectural choice of the NTMs is the type of the neural network that would be used as a controller. More precisely, we have to decide whether we want to use feedforward or recurrent network. 
"If one compares the controller to the central processing unit in a digital computer (albeit with adaptive rather than predefined instructions) and the memory matrix to RAM, then the hidden activations of the recurrent controller are akin to the registers in the processor"
It is interesting that if we choose feedforward networks as a controller we can mimic RNNs by reading and writing at the same location at every time step.

Controller Network

In Neural Turing Machines (NTMs), components are differentiable because the read and write operations are 'blurry.' Instead of addressing a single memory element like a standard Turing machine or digital computer, NTM operations interact to varying degrees with all elements in the memory. An attentional focus mechanism determines this degree of blurriness, limiting input and output interactions to relatively small parts of the memory while neglecting the rest.

Learn Before

Related

Learn After