Within a sequence-to-sequence encoder, an RNN processes the input sequence one token at a time. At each time step $$t$$, the recurrent layer applies a transformation function $$f$$ that combines the input feature vector $$\mathbf{x}_t$$ (derived from the $$t^{\textrm{th}}$$ token $$x_t$$) with the hidden state $$\mathbf{h}_{t-1}$$ carried from the preceding time step to produce the updated hidden state $$\mathbf{h}_t$$:

$$\mathbf{h}_t = f(\mathbf{x}_t, \mathbf{h}_{t-1})$$

This recurrence captures how the encoder incrementally builds a representation of the input sequence, with each hidden state $$\mathbf{h}_t$$ encoding information about all tokens observed up to and including position $$t$$.

RNN Encoder Hidden State Recurrence

The `Seq2SeqEncoder` class implements the RNN-based encoder for sequence-to-sequence learning by extending a base `Encoder` interface. Its architecture consists of two primary components: an **embedding layer** that converts each input token index into a dense feature vector, and a **multilayer GRU** that processes the resulting sequence of embeddings. The embedding layer's weight matrix has a shape of (`vocab_size`, `embed_size`), where each row $$i$$ stores the feature vector for the token with index $$i$$. During the forward pass, the input tensor of shape (batch_size, num_steps) is first transposed and embedded to produce a tensor of shape (num_steps, batch_size, embed_size). The GRU then processes this sequence and returns two outputs: `outputs` of shape (num_steps, batch_size, num_hiddens), containing the final-layer hidden states at every time step, and `state` of shape (num_layers, batch_size, num_hiddens), containing the hidden states of all layers at the final time step. All weights are initialized using Xavier initialization.

Seq2SeqEncoder Implementation

In the encoder component of a sequence-to-sequence model, the choice between a unidirectional and a bidirectional RNN determines how much input context each hidden state can capture. A **unidirectional RNN encoder** computes hidden states that depend solely on the input subsequence at and before the current time step, meaning each $$\mathbf{h}_t$$ encodes only the tokens $$x_1, \ldots, x_t$$. In contrast, a **bidirectional RNN encoder** produces hidden states that also incorporate information from tokens appearing after the current position, effectively encoding the information of the entire input sequence at every time step. While a bidirectional encoder provides richer context, the unidirectional design is simpler and sufficient for many seq2seq applications.

Unidirectional vs. Bidirectional Encoder in Seq2Seq

In an encoder–decoder architecture, the encoder is designed to process input data that may vary in size. It accepts a variable-length sequence as its input and mathematically transforms it into an encoded state that has a fixed, predetermined shape. This fixed-shape state acts as a compressed summary containing the crucial context from the original sequence. In a common approach, the encoder uses an RNN to compute hidden states $$\mathbf{h}_1, \ldots, \mathbf{h}_T$$ for all time steps and then derives the context variable $$\mathbf{c}$$ through a customized function $$q$$: $$\mathbf{c} = q(\mathbf{h}_1, \ldots, \mathbf{h}_T)$$. A simple choice for $$q$$ is to set $$\mathbf{c} = \mathbf{h}_T$$, using the final hidden state as the complete context.

Claude

In object-oriented implementations of the encoder-decoder architecture, the base encoder acts as an interface rather than a concrete model. It defines the structural blueprint by specifying that the encoder must accept variable-length sequences, denoted as $$X$$, as its input. This base class does not define the internal computational logic; instead, any specific neural network model (such as a Recurrent Neural Network or a Transformer) that inherits from this base `Encoder` class is required to provide its own concrete implementation for the forward pass.

Encoder Base Interface

Dive into Deep Learning

Encoder Transformation to a Fixed-Shape State

The Decoder serves as the abstract base interface for the decoding phase within the encoder-decoder architecture. To support the step-by-step generation of variable-length sequences, this interface defines two essential methods. First, the init_state method converts the outputs generated by the encoder (and potentially additional information, such as the valid sequence length) into the initial internal state of the decoder. Second, a forward execution method (such as forward in PyTorch, __call__ in JAX, or call in TensorFlow) is mandated; this method maps the input at the current time step (often the token generated at the previous step) and the current internal state to produce the output token for the current time step.

Learn Before

Related

Learn After