This is the third type of multiplicative attention. Here you can see the formula of that:

$ score(h, h'_{t}) = V^{T}_{a} * tanh( W_{a} * [h, h'_{t}] )$

$h$ - encoder vector
$h'_{t}$ - decoder vector 

So first we concatenate the encoder and decoder state. And we just add a usual Dense layer with tanh activation to the input and also add a layer with one unit to represent the score. So as we train this mechanism actually learn which words are most influential for the output words. 

Concat Attention Function

This is the first type of multiplicative attention. This is the easiest option to implement and it is based on dot similarity. Geometrically, the dot product depends on the angle in between the vectors. The intuition behind it is that if vectors have similar direction then dot product will be higher and further the direction goes smaller the dot similarly will become. For example dot product of orthogonal vectors will be zero. And in this case the encoder vectors that are more related to the decoder vector will get bigger scores. One disadvantage for this is that it does not use any learning while other types utilize some kind of learning:

$ score(h, h'_{t}) = (h'_{t})^{T} *  h$. 

$h$ - encoder
$h'_{t}$ - decoder

Dot Attention

This is similar to dot attention but also adds a learning aspect to it because we first pass the encoder vector through a Dense layer before taking the dot product. In this case attention is also is subject to backpropagation and gradient descent

$ score(h, h'_{t}) = (h'_{t})^{T} * W_{a} * h $

$h$ - encoder state
$h'_{t}$ - decoder state

General Attention function

It is a bit different from Multiplicative Concat score but also uses a MultiLayer perceptron:

$ score(h, h'_{t-1}) = V_{a}^{T} * tanh( W^{T} h'_{t-1} + U^{T} h) $

$h$ - encoder vector
$h'_{t-1}$ - decoder previous vector

It is very similar to concat function but does not concatenate the input states but we concat them later after we apply different two Tanh Dense Layers separately to encoder and decoder states. After we concat them here we also add a singular Dense layer to represent the score. We also use here previous decoder state rather than current in Multiplicative attention

Additive Attention Function

There are major two types of attention - Those are Additive and Multiplicative attention. Those are also called Bahdanau and Luong attention based on the first authors of two papers introducing those methods:

- Additive (Bahdanau) in here we actually use previous decoder hidden state to calculate score for the next decoder state

- Multiplicative(Luong) - this one uses current decoder state to calculate score for the each of the decoder states


University of California, Berkeley

The most upgrades are here. The RNN cell is left untouched but we use the encoder hidden states to generate words based on decoder hidden states. So the word we generate is a function of all encoder hidden states and current decoder state. But for inputs of different length, we are going to have different amounts of encoder hidden units. How do we fix the input dimension for that output function. One way is to add weight to each encoder hidden state based on its connection to the current decoder state and add them up. So for each hidden state we will define a score function:

Score( encoder hidden_state, decoder_hidden_state)

Encoder States: $h_{1} , h_{2}, h_{3}$  with  Decoder current state: $h'_{t}$

Scores: $ score(h_{1}, h'_{t}), score(h_{2}, h'_{t}), score(h_{3}, h'_{t}) $

Softmax: $s_{1} , s_{2}, s_{3}$ 

Context vector $ c_{t} = s_{1}  * h_{1} + s_{2} * h_{2} + s_{3} * h_{3} $

Over here you can see how the scoring function will work for each of the decoder states. As you can see here we score each encoder state and then apply a softmax function on it in order to normalize the scores and then we add them up. Then this context vector will be used further along with the decoder state to determine current word 


Attention Decoder

One of the main papers that introduces the attention mechanism 

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.

Neural Machine Translation by Jointly Learning to Align and Translate

Also another main papers on attention mechanism
Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.

Learn Before

Related

Learn After