1. We can choose weighting by content-based system without any modification
2. We can choose weighting by content-based system and apply shifting 
3. "Weighting from the previous time step can be rotated without any input from the content-based addressing system"

San Diego State University

The weights are produced using two addressing mechanisms that are:
1. Content-based 
2. Location-based 

Addressing Mechanisms (NTM Architecture)

https://arxiv.org/pdf/1410.5401.pdf?fbclid=IwAR1mft01ZW2dGNGyVczj4HR6E5H8xKpqTlniZeuI8JxGuSNzOdoNupq4j9M

Neural Turing Machines - Original Paper Reference

Here we focus on location based on the similarity between the values that were emitted by the controller.
Here both read/write firstly focuses on a length M key vector $K_t$ and we compare them to $M_t(i)$ vectors. This method produces normalized weights:
$$w_t^c(i) = \frac {exp(\beta_t K[k_t, M_t(i)])} {\sum_j \beta_t K[k_t, M_t(j)]}$$
The similarity is measured using cosine similarity:
$$k[u, v] = \frac {u v} {||u||  ||v||} $$


Content-based (NTM Architecture)

In case, content-based method isn't well-suited for the problem, location-based addressing is also used which purely focuses on the location.

Location-based addressing uses rotational shifting and weighting.
Before rotational shifting will be performed, both read/write heads produce scalar interpolation gate $g_t$ (between 0 and 1). This value acts like a blend between the $w_{t-1}$ (weight produced in the previous time step by read/write head)  and the $w_t^c$ generated by the content-based system and as the result gated weight
$w_t^g$ is returned:
$$w_t^g \leftarrow g_t w_t^c + (1 - g_t) w_{t-1}$$

Depending on the value of g we might completely ignore weights produced by content-based system or by head in the previous timestamp. More precisely, if g is equal to we ignore content-based system and if it is 1 we ignore head.

After this procedure, shift weighting $s_t$ is applied. The simplest way would be to use softmax for defining shift weighting. Rotation that is applied to our gated weight is written in the following formula:
$$w_t^{\sim} \leftarrow \sum_{j=i}^{N-1} w_t^g (j) s_t(i-j)$$ 
If weights aren't sharp this convolution procedure can lead to leakage or dispersion. In order to solve this problem, each head produces additional scalar $\gamma_t \ge 1$ which makes sure to sharpen those weights:
$$w_t(i)  \leftarrow \frac {w_t^{\sim} (i)^{\gamma_t}}{\sum_jw_t^{\sim}(j)^{\gamma_t}}$$

Learn Before

Related