Within the encoder–decoder framework, the decoder is responsible for producing the final output based on the information gathered by the encoder. It maps the fixed-shape encoded state back into a new, variable-length sequence. This capability to transition from a fixed-shape context to a flexible output length is what makes the architecture suitable for sequence-to-sequence problems such as machine translation.

Decoder Mapping from a Fixed-Shape State

The AttentionDecoder class establishes the abstract base interface for all attention-based decoders by extending the standard Decoder base class. It inherits the init_state and forward-pass methods from its parent while introducing an additional contract: an attention_weights property. Subclasses must override this property to return the attention weight distributions computed during decoding. This added requirement ensures that any concrete attention-based decoder provides access to its internal attention weights, which is essential for interpretability and visualization of where the model focuses in the source sequence at each decoding step.

AttentionDecoder Base Interface

The Decoder serves as the abstract base interface for the decoding phase within the encoder-decoder architecture. To support the step-by-step generation of variable-length sequences, this interface defines two essential methods. First, the init_state method converts the outputs generated by the encoder (and potentially additional information, such as the valid sequence length) into the initial internal state of the decoder. Second, a forward execution method (such as forward in PyTorch, __call__ in JAX, or call in TensorFlow) is mandated; this method maps the input at the current time step (often the token generated at the previous step) and the current internal state to produce the output token for the current time step.

Claude

In object-oriented implementations of the encoder-decoder architecture, the base encoder acts as an interface rather than a concrete model. It defines the structural blueprint by specifying that the encoder must accept variable-length sequences, denoted as $$X$$, as its input. This base class does not define the internal computational logic; instead, any specific neural network model (such as a Recurrent Neural Network or a Transformer) that inherits from this base `Encoder` class is required to provide its own concrete implementation for the forward pass.

Encoder Base Interface

Dive into Deep Learning

In an encoder–decoder architecture, the encoder is designed to process input data that may vary in size. It accepts a variable-length sequence as its input and mathematically transforms it into an encoded state that has a fixed, predetermined shape. This fixed-shape state acts as a compressed summary containing the crucial context from the original sequence. In a common approach, the encoder uses an RNN to compute hidden states $$\mathbf{h}_1, \ldots, \mathbf{h}_T$$ for all time steps and then derives the context variable $$\mathbf{c}$$ through a customized function $$q$$: $$\mathbf{c} = q(\mathbf{h}_1, \ldots, \mathbf{h}_T)$$. A simple choice for $$q$$ is to set $$\mathbf{c} = \mathbf{h}_T$$, using the final hidden state as the complete context.

Learn Before

Related

Learn After