Within a sequence-to-sequence decoder, the RNN updates its hidden state at each time step $$t'$$ using a transformation function $$g$$ that takes three inputs: the previous target token $$y_{t'-1}$$, the encoder's context variable $$\mathbf{c}$$, and the decoder's hidden state from the preceding time step $$\mathbf{s}_{t'-1}$$. The resulting hidden state update is expressed as:

$$\mathbf{s}_{t'} = g(y_{t'-1}, \mathbf{c}, \mathbf{s}_{t'-1})$$

This formulation mirrors the encoder's recurrence but differs in a key way: the decoder's transformation incorporates the context variable $$\mathbf{c}$$ as an additional input at every time step, ensuring that the encoded source sequence information continuously influences the generation process.

RNN Decoder Hidden State Recurrence

Within the encoder–decoder framework, the decoder is responsible for producing the final output based on the information gathered by the encoder. It maps the fixed-shape encoded state back into a new, variable-length sequence. This capability to transition from a fixed-shape context to a flexible output length is what makes the architecture suitable for sequence-to-sequence problems such as machine translation.

Claude

The Decoder serves as the abstract base interface for the decoding phase within the encoder-decoder architecture. To support the step-by-step generation of variable-length sequences, this interface defines two essential methods. First, the init_state method converts the outputs generated by the encoder (and potentially additional information, such as the valid sequence length) into the initial internal state of the decoder. Second, a forward execution method (such as forward in PyTorch, __call__ in JAX, or call in TensorFlow) is mandated; this method maps the input at the current time step (often the token generated at the previous step) and the current internal state to produce the output token for the current time step.

Decoder Base Interface

Dive into Deep Learning

Decoder Mapping from a Fixed-Shape State

The AttentionDecoder class establishes the abstract base interface for all attention-based decoders by extending the standard Decoder base class. It inherits the init_state and forward-pass methods from its parent while introducing an additional contract: an attention_weights property. Subclasses must override this property to return the attention weight distributions computed during decoding. This added requirement ensures that any concrete attention-based decoder provides access to its internal attention weights, which is essential for interpretability and visualization of where the model focuses in the source sequence at each decoding step.

Learn Before

Related

Learn After