This is the third type of multiplicative attention. Here you can see the formula of that:

$ score(h, h'_{t}) = V^{T}_{a} * tanh( W_{a} * [h, h'_{t}] )$

$h$ - encoder vector
$h'_{t}$ - decoder vector 

So first we concatenate the encoder and decoder state. And we just add a usual Dense layer with tanh activation to the input and also add a layer with one unit to represent the score. So as we train this mechanism actually learn which words are most influential for the output words. 

Concat Attention Function

This is similar to dot attention but also adds a learning aspect to it because we first pass the encoder vector through a Dense layer before taking the dot product. In this case attention is also is subject to backpropagation and gradient descent

$ score(h, h'_{t}) = (h'_{t})^{T} * W_{a} * h $

$h$ - encoder state
$h'_{t}$ - decoder state

General Attention function

Dot product attention is a fundamental type of multiplicative attention based on dot similarity. Geometrically, the dot product measures the alignment between vectors: if a query and key share a similar direction, their dot product is higher, whereas orthogonal vectors yield a dot product of $$0$$. This implies that keys which are more conceptually related to the current query will receive larger attention scores. One notable characteristic of pure dot product attention is that it does not introduce any additional learnable parameters, relying entirely on the existing vector representations. The attention score is calculated mathematically using the standard query-key notation as: $$ a(\mathbf{q}, \mathbf{k}_i) = \mathbf{q}^	op \mathbf{k}_i $$ where $$\mathbf{q}$$ represents the query vector and $$\mathbf{k}_i$$ represents the key vector.

Dot Product Attention

Additive attention is a scoring function designed for situations where queries and keys reside in vector spaces of differing dimensionality, making a direct dot product infeasible. Introduced by Bahdanau et al. (2014), it projects both the query $$\mathbf{q} \in \mathbb{R}^q$$ and the key $$\mathbf{k} \in \mathbb{R}^k$$ into a shared hidden space of dimension $$h$$ using separate learned weight matrices $$\mathbf{W}_q \in \mathbb{R}^{h 	imes q}$$ and $$\mathbf{W}_k \in \mathbb{R}^{h 	imes k}$$. These projections are summed element-wise and passed through a $$	anh$$ nonlinearity, after which a learned weight vector $$\mathbf{w}_v \in \mathbb{R}^h$$ reduces the result to a scalar attention score:

$$a(\mathbf{q}, \mathbf{k}) = \mathbf{w}_v^	op 	anh(\mathbf{W}_q \mathbf{q} + \mathbf{W}_k \mathbf{k}) \in \mathbb{R}$$

This score is subsequently fed into a softmax function to produce nonnegative, normalized attention weights. An equivalent interpretation views additive attention as concatenating the query and key and feeding them through an MLP with a single hidden layer using $$	anh$$ activation. As its name suggests, the computation is additive rather than multiplicative, which can yield minor computational savings compared to approaches that require matrix products between query and key vectors.

Additive Attention Scoring Function

There are major two types of attention - Those are Additive and Multiplicative attention. Those are also called Bahdanau and Luong attention based on the first authors of two papers introducing those methods:

- Additive (Bahdanau) in here we actually use previous decoder hidden state to calculate score for the next decoder state
- Multiplicative(Luong) - this one uses current decoder state to calculate score for the each of the decoder states

Claude

University of California, Berkeley

In an attention-based decoder, the RNN cell itself remains unchanged, but the encoder's hidden states are leveraged to inform word generation at each decoder time step. The word produced at each step becomes a function of every encoder hidden state together with the current decoder state. Because input sequences can vary in length, the number of encoder hidden states differs across examples, posing a challenge for fixing the input dimension of the output function. This is resolved by assigning a learned weight to each encoder hidden state based on its relevance to the current decoder state, then summing the weighted states. A score function evaluates the compatibility between each encoder hidden state and the decoder state:

Encoder states: $$h_{1}, h_{2}, h_{3}$$

Decoder current state: $$h'_{t}$$

Scores: $$\text{score}(h_{1}, h'_{t}),\; \text{score}(h_{2}, h'_{t}),\; \text{score}(h_{3}, h'_{t})$$

Applying softmax yields normalized weights: $$s_{1}, s_{2}, s_{3}$$

The context vector is computed as: $$c_{t} = s_{1} \cdot h_{1} + s_{2} \cdot h_{2} + s_{3} \cdot h_{3}$$

Each encoder state is scored against the decoder state, the scores are normalized via softmax, and the resulting weighted sum forms a context vector. This context vector, combined with the decoder hidden state, determines the current output word. In the Bahdanau attention variant specifically, the decoder hidden state at the previous time step serves as the query, and the encoder hidden states at all time steps serve as both the keys and the values.

Attention Decoder

One of the main papers that introduces the attention mechanism. Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. https://arxiv.org/abs/1409.0473

Neural Machine Translation by Jointly Learning to Align and Translate

Also another main papers on attention mechanism
Luong, M. T., Pham, H., & Manning, C. D. (2015). Effective approaches to attention-based neural machine translation. arXiv preprint arXiv:1508.04025.

Effective Approaches to Attention-based Neural Machine Translation

Dive into Deep Learning

Attention Scoring Functions

The AttentionDecoder class establishes the abstract base interface for all attention-based decoders by extending the standard Decoder base class. It inherits the init_state and forward-pass methods from its parent while introducing an additional contract: an attention_weights property. Subclasses must override this property to return the attention weight distributions computed during decoding. This added requirement ensures that any concrete attention-based decoder provides access to its internal attention weights, which is essential for interpretability and visualization of where the model focuses in the source sequence at each decoding step.

AttentionDecoder Base Interface

The Seq2SeqAttentionDecoder class implements a concrete RNN decoder that integrates Bahdanau-style additive attention into the sequence-to-sequence framework. Its architecture consists of four components: an AdditiveAttention module, an embedding layer that maps target token indices to dense vectors, a multilayer GRU whose input size equals embed_size + num_hiddens (to accommodate the concatenated context and embedding), and a fully connected output layer that projects hidden states to the target vocabulary size. All parameters are initialized using Xavier initialization.

Learn Before

Related

Learn After