1Cademy - Generating Sequence Representations with a Pre-trained Encoder

Learn Before

Fine-tuning for Sequence Encoding Models

Activity (Process)

Generating Sequence Representations with a Pre-trained Encoder

A pre-trained sequence encoding model, with its parameters optimized to $\hat{\theta}$ , transforms an input sequence of tokens, $x = \{x_0, x_1, ..., x_m\}$ , into a numerical representation, $H$ . This output, $H$ , is a sequence of real-valued vectors, { $h_0, h_1, ..., h_m\}, where each vector$ h_i $represents the token$ x_i $in its context. The entire output$ H $can be structured as a matrix by treating each vector$ h_i$ as a row: $\mathbf{H} = \begin{bmatrix} \mathbf{h}_0 \\ \vdots \\ \mathbf{h}_m \end{bmatrix}$ The specific equation for this transformation is defined separately.