Here is the flow of the additive attention function in terms of Neural Networks in order to give an intuition on how to implement it in Keras and Tensorflow



University of California, Berkeley

It is a bit different from Multiplicative Concat score but also uses a MultiLayer perceptron:

$ score(h, h'_{t-1}) = V_{a}^{T} * tanh( W^{T} h'_{t-1} + U^{T} h) $

$h$ - encoder vector
$h'_{t-1}$ - decoder previous vector

It is very similar to concat function but does not concatenate the input states but we concat them later after we apply different two Tanh Dense Layers separately to encoder and decoder states. After we concat them here we also add a singular Dense layer to represent the score. We also use here previous decoder state rather than current in Multiplicative attention

Learn Before

Related