Google

An encoder, denoted as $$\mathrm{Encoder}_{\theta}(\cdot)$$, is a function parameterized by $$\theta$$ that processes an input sequence of tokens, $$\mathbf{x}=x_0...x_m$$, to produce a corresponding sequence of output vectors, $$\mathbf{H}=\mathbf{h}_0...\mathbf{h}_m$$. If each $$\mathbf{h}_i$$ is viewed as a row vector, the entire output sequence can be represented as the matrix $$\mathbf{H} = \begin{bmatrix} \mathbf{h}_0 \\ \vdots \\ \mathbf{h}_m \end{bmatrix}$$.

Input and Output of a Sequence Encoder

A model processes the input sentence 'The cat sat.' which is broken down into a sequence of 4 tokens: ['The', 'cat', 'sat', '.']. If this model functions as a sequence encoder, what is the most accurate description of the output it generates?

A developer is building a system for part-of-speech tagging, a task that requires assigning a grammatical label (e.g., 'noun', 'verb') to every single word in an input sentence. They need to choose a model architecture that produces a suitable numerical representation to accomplish this. Analyze the two options below and determine which is appropriate for the task, justifying your reasoning.

Model Output for a Token-Level Task

A sequence encoder processes an input sequence of 10 tokens and produces a single, fixed-size vector that represents the entire sequence's meaning.

A text classification system can be constructed by placing a neural network classifier on top of an encoder. If the classifier is denoted as $$ \mathrm{Classify}_{\omega}(\cdot) $$ with parameters $$ \omega $$, the complete probabilistic text classification model is mathematically represented as: $$ \mathrm{Pr}_{\omega,\hat{\theta}}(\cdot|\mathbf{x}) = \mathrm{Classify}_{\omega}(\mathbf{H}) = \mathrm{Classify}_{\omega}(\mathrm{Encode}_{\hat{\theta}}(\mathbf{x})) $$. In this equation, $$ \mathbf{x} $$ denotes the input sequence, and $$ \mathbf{H} $$ is the numerical representation produced by the encoder. The term $$ \mathrm{Pr}_{\omega,\hat{\theta}}(\cdot|\mathbf{x}) $$ defines a probability distribution across a predetermined set of labels, and the system's final output is the specific label that achieves the highest probability within this distribution.

Probabilistic Model for Text Classification using an Encoder-Classifier Architecture

Training an encoder model is not straightforward because there is no naturally occurring, gold-standard data available to directly measure the quality of the real-valued function's output vectors.

Challenge of Encoder Pre-training Evaluation

To overcome the lack of direct supervision signals in encoder pre-training, a typical approach is to combine the encoder with output layers that generate easier-to-obtain supervision signals. For instance, adding a Softmax layer on top of a Transformer encoder creates an architecture identical to a decoder-based language model, allowing the system to output a sequence of probability distributions for training.

Learn Before

Related