Learn Before
Formula

Concat Attention Scoring Function

The concat (concatenation) attention scoring function is one of the three attention scoring functions proposed by Luong et al. (2015). It calculates the alignment score between an encoder hidden state h\mathbf{h} and a decoder hidden state ht\mathbf{h}'_t as:

score(h,ht)=vaTtanh(Wa[h;ht])\text{score}(\mathbf{h}, \mathbf{h}'_t) = \mathbf{v}_a^T \tanh(\mathbf{W}_a [\mathbf{h}; \mathbf{h}'_t])

where:

  • h\mathbf{h} is the encoder vector (source hidden state).
  • ht\mathbf{h}'_t is the decoder vector (target hidden state at time step tt).
  • Wa\mathbf{W}_a and vaT\mathbf{v}_a^T are learnable weight parameters.
  • [h;ht][\mathbf{h}; \mathbf{h}'_t] represents the concatenation of the encoder and decoder states.

In a neural network, this is implemented by concatenating the encoder and decoder states, applying a dense layer with a tanh\tanh activation, and projecting the result to a single scalar to produce the final score. During training, this mechanism learns which source words are most influential for generating target words.

0

1

Updated 2026-06-18

Tags

Data Science