The Seq2SeqAttentionDecoder class implements a concrete RNN decoder that integrates Bahdanau-style additive attention into the sequence-to-sequence framework. Its architecture consists of four components: an AdditiveAttention module, an embedding layer that maps target token indices to dense vectors, a multilayer GRU whose input size equals embed_size + num_hiddens (to accommodate the concatenated context and embedding), and a fully connected output layer that projects hidden states to the target vocabulary size. All parameters are initialized using Xavier initialization.

Seq2SeqAttentionDecoder Implementation

The AttentionDecoder class establishes the abstract base interface for all attention-based decoders by extending the standard Decoder base class. It inherits the init_state and forward-pass methods from its parent while introducing an additional contract: an attention_weights property. Subclasses must override this property to return the attention weight distributions computed during decoding. This added requirement ensures that any concrete attention-based decoder provides access to its internal attention weights, which is essential for interpretability and visualization of where the model focuses in the source sequence at each decoding step.

Claude

The Decoder serves as the abstract base interface for the decoding phase within the encoder-decoder architecture. To support the step-by-step generation of variable-length sequences, this interface defines two essential methods. First, the init_state method converts the outputs generated by the encoder (and potentially additional information, such as the valid sequence length) into the initial internal state of the decoder. Second, a forward execution method (such as forward in PyTorch, __call__ in JAX, or call in TensorFlow) is mandated; this method maps the input at the current time step (often the token generated at the previous step) and the current internal state to produce the output token for the current time step.

Decoder Base Interface

In an attention-based decoder, the RNN cell itself remains unchanged, but the encoder's hidden states are leveraged to inform word generation at each decoder time step. The word produced at each step becomes a function of every encoder hidden state together with the current decoder state. Because input sequences can vary in length, the number of encoder hidden states differs across examples, posing a challenge for fixing the input dimension of the output function. This is resolved by assigning a learned weight to each encoder hidden state based on its relevance to the current decoder state, then summing the weighted states. A score function evaluates the compatibility between each encoder hidden state and the decoder state:

Encoder states: $$h_{1}, h_{2}, h_{3}$$

Decoder current state: $$h'_{t}$$

Scores: $$\text{score}(h_{1}, h'_{t}),\; \text{score}(h_{2}, h'_{t}),\; \text{score}(h_{3}, h'_{t})$$

Applying softmax yields normalized weights: $$s_{1}, s_{2}, s_{3}$$

The context vector is computed as: $$c_{t} = s_{1} \cdot h_{1} + s_{2} \cdot h_{2} + s_{3} \cdot h_{3}$$

Each encoder state is scored against the decoder state, the scores are normalized via softmax, and the resulting weighted sum forms a context vector. This context vector, combined with the decoder hidden state, determines the current output word. In the Bahdanau attention variant specifically, the decoder hidden state at the previous time step serves as the query, and the encoder hidden states at all time steps serve as both the keys and the values.

Attention Decoder

Dive into Deep Learning

Within the encoder–decoder framework, the decoder is responsible for producing the final output based on the information gathered by the encoder. It maps the fixed-shape encoded state back into a new, variable-length sequence. This capability to transition from a fixed-shape context to a flexible output length is what makes the architecture suitable for sequence-to-sequence problems such as machine translation.

Decoder Mapping from a Fixed-Shape State

AttentionDecoder Base Interface

There are major two types of attention - Those are Additive and Multiplicative attention. Those are also called Bahdanau and Luong attention based on the first authors of two papers introducing those methods:

- Additive (Bahdanau) in here we actually use previous decoder hidden state to calculate score for the next decoder state
- Multiplicative(Luong) - this one uses current decoder state to calculate score for the each of the decoder states

Learn Before

Related

Learn After