Here is the flow of the concat attention function in terms of Neural Network layers that gives an idea of how to implement it in Keras or Tensorflow

University of California, Berkeley

This is the third type of multiplicative attention. Here you can see the formula of that:

$ score(h, h'_{t}) = V^{T}_{a} * tanh( W_{a} * [h, h'_{t}] )$

$h$ - encoder vector
$h'_{t}$ - decoder vector 

So first we concatenate the encoder and decoder state. And we just add a usual Dense layer with tanh activation to the input and also add a layer with one unit to represent the score. So as we train this mechanism actually learn which words are most influential for the output words. 

Learn Before

Related