Here is the flow of the additive attention function in terms of Neural Networks in order to give an intuition on how to implement it in Keras and Tensorflow



The Flow of the Additive Attention Function

It is a bit different from Multiplicative Concat score but also uses a MultiLayer perceptron:

$ score(h, h'_{t-1}) = V_{a}^{T} * tanh( W^{T} h'_{t-1} + U^{T} h) $

$h$ - encoder vector
$h'_{t-1}$ - decoder previous vector

It is very similar to concat function but does not concatenate the input states but we concat them later after we apply different two Tanh Dense Layers separately to encoder and decoder states. After we concat them here we also add a singular Dense layer to represent the score. We also use here previous decoder state rather than current in Multiplicative attention

University of California, Berkeley

There are major two types of attention - Those are Additive and Multiplicative attention. Those are also called Bahdanau and Luong attention based on the first authors of two papers introducing those methods:

- Additive (Bahdanau) in here we actually use previous decoder hidden state to calculate score for the next decoder state

- Multiplicative(Luong) - this one uses current decoder state to calculate score for the each of the decoder states


Attention Scoring Functions

One of the main papers that introduces the attention mechanism 

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

Neural Machine Translation by Jointly Learning to Align and Translate

This is the third type of multiplicative attention. Here you can see the formula of that:

$ score(h, h'_{t}) = V^{T}_{a} * tanh( W_{a} * [h, h'_{t}] )$

$h$ - encoder vector
$h'_{t}$ - decoder vector 

So first we concatenate the encoder and decoder state. And we just add a usual Dense layer with tanh activation to the input and also add a layer with one unit to represent the score. So as we train this mechanism actually learn which words are most influential for the output words. 

Concat Attention Function

This is the first type of multiplicative attention. This is the easiest option to implement and it is based on dot similarity. Geometrically, the dot product depends on the angle in between the vectors. The intuition behind it is that if vectors have similar direction then dot product will be higher and further the direction goes smaller the dot similarly will become. For example dot product of orthogonal vectors will be zero. And in this case the encoder vectors that are more related to the decoder vector will get bigger scores. One disadvantage for this is that it does not use any learning while other types utilize some kind of learning:

$ score(h, h'_{t}) = (h'_{t})^{T} *  h$. 

$h$ - encoder
$h'_{t}$ - decoder

Dot Attention

This is similar to dot attention but also adds a learning aspect to it because we first pass the encoder vector through a Dense layer before taking the dot product. In this case attention is also is subject to backpropagation and gradient descent

$ score(h, h'_{t}) = (h'_{t})^{T} * W_{a} * h $

$h$ - encoder state
$h'_{t}$ - decoder state

Learn Before

Related

Learn After