Google

Diverse NLP problems can be unified under a general sequence model structure, represented by the function $$\mathbf{o} = g(x_0, x_1, ..., x_m; \theta)$$. In this formula, $$x_0, x_1, ..., x_m$$ is the input token sequence, where $$x_0$$ is a special start-of-sequence symbol (such as $$\langle s \rangle$$ or $$\mathbf{[\mathrm{CLS}]}$$). The function $$g(\cdot; \theta)$$ (also written as $$g_{\theta}(\cdot)$$) is a neural network defined by parameters $$\theta$$, and $$\mathbf{o}$$ is the model's output. A common shorthand for the output is $$\mathbf{o} = g_{\theta}(x_0, x_1, ..., x_m)$$.

General Formulation of a Sequence Model

The output $$\mathbf{o}$$ from a general sequence model, which is generated by a neural network $$g(\cdot; \theta)$$, can differ based on the specific problem being addressed. For token prediction problems (such as language modeling), the output $$\mathbf{o}$$ is typically a probability distribution over a defined vocabulary. Conversely, for sequence encoding problems, the output $$\mathbf{o}$$ serves as a representation of the input sequence, commonly expressed as a sequence of real-valued vectors.

Output Variation in Sequence Models

The special $$[\mathrm{CLS}]$$ token is typically prepended as the start symbol, denoted as $$x_0$$, of an input sequence before it enters an encoder. While it marks the start at the input level, its primary utility is at the output stage. The encoder's final hidden state corresponding to this token, denoted as $$\mathbf{h}_0$$, is generally considered to be the representation of the entire sequence. Consequently, this aggregate vector is often fed into a classification layer, such as a Softmax layer, to construct systems for sequence-level tasks like binary classification.

Role of the [CLS] Token in Sequence Classification

Masked Language Modeling (MLM) is a highly popular pre-training method for encoders and forms the foundation for models like BERT. The core principle involves creating a prediction task by masking some tokens in an input sequence. The model is then trained to predict these original, masked tokens by leveraging the surrounding unmasked tokens as context. This process forces the model to develop a deep, bidirectional understanding of language by considering both left and right contexts.

Masked Language Modeling

For tasks involving multiple text segments, a common input formatting strategy is to concatenate the sequences and use a special separator token, like `[SEP]`, to distinguish between them. For instance, an input sequence $x_m$ and a target sequence $y_1, y_2, \dots, y_n$ can be combined into a single input stream for a model as: $x_m$ `[SEP]` $y_1 y_2 \dots y_n$ `[SEP]`.

Input Formatting with Separator Tokens

In standard auto-regressive language models, the joint probability of a token sequence is factored using the chain rule of probability. In neural network implementations, this conditioning on previous tokens is practically achieved by using their embeddings. This relationship for a sequence $\mathbf{x} = (x_0, ..., x_4)$ can be expressed with the following formula, which shows the equivalence between the probabilistic formulation and its neural network counterpart: $$ \text{Pr}(\mathbf{x}) = \text{Pr}(x_0) \cdot \text{Pr}(x_1|x_0) \cdot \text{Pr}(x_2|x_0, x_1) \cdot \text{Pr}(x_3|x_0, x_1, x_2) \cdot \text{Pr}(x_4|x_0, x_1, x_2, x_3) \\ = \text{Pr}(x_0) \cdot \text{Pr}(x_1|\mathbf{e}_0) \cdot \text{Pr}(x_2|\mathbf{e}_0, \mathbf{e}_1) \cdot \text{Pr}(x_3|\mathbf{e}_0, \mathbf{e}_1, \mathbf{e}_2) \cdot \text{Pr}(x_4|\mathbf{e}_0, \mathbf{e}_1, \mathbf{e}_2, \mathbf{e}_3) $$ where $\mathbf{e}_i$ represents the embedding of token $x_i$.

Standard Auto-Regressive Probability Factorization using Embeddings

In encoder pre-training and related tasks, the special $$[\mathrm{CLS}]$$ token is conventionally used as the start symbol for an input sequence. It is typically the first token, denoted as $$x_0$$, and serves as a generic start-of-sequence marker.

CLS Token as a Start Symbol in Encoder Pre-training

Causal Language Modeling (CLM) and Masked Language Modeling (MLM) differ fundamentally in how they use context for token prediction. CLM is unidirectional, meaning it only uses the left-context (preceding tokens) to predict the token at a given position. In contrast, MLM is bidirectional, as it utilizes all unmasked tokens—both to the left and right of a masked token—to make its prediction. This bidirectional approach allows MLM-based models to build a more comprehensive contextual understanding of language.

Comparison of Context Usage in Causal vs. Masked Language Modeling

A software company is developing a system to automatically classify user feedback. The system uses a model based on the general formulation `o = g(x_0, x_1, ..., x_m; θ)`. Your task is to analyze how this formula applies to the specific case provided below. Explain what each of the three main components—the input sequence `(x_0, x_1, ..., x_m)`, the function `g(·; θ)`, and the output `o`—would represent in this context.

Applying the General Sequence Model Formulation

In the general formulation of a sequence model, `o = g(x_0, x_1, ..., x_m; θ)`, which statement best analyzes the distinct roles of the components?

Match each symbol from the general sequence model formulation, `o = g(x_0, x_1, ..., x_m; θ)`, with its correct description.

While the general formulation of a sequence model provides a unifying framework, it is accompanied by at least two fundamental challenges. These issues are inherent to the formulation and require consideration in its practical application.

Fundamental Issues in Sequence Model Formulation

In a sequence model, the function denoted by $$g(\cdot; \theta)$$ (also written as $$g_{\theta}(\cdot)$$) represents a neural network. The symbol $$\theta$$ corresponds to the parameters that define the neural network.

Learn Before

Related