Google

Pre-training in Natural Language Processing primarily addresses two categories of problems: sequence modeling, also referred to as sequence encoding, and sequence generation. Despite their distinct forms, these two problem types can be conceptually unified and described using a single, general model formulation for the sake of simplicity.

Core Problem Types in NLP Pre-training

A Sequence Encoding Model is a type of model that processes a sequence of words or tokens and transforms it into a numerical representation. This output, which can take the form of a single vector or a sequence of vectors, encapsulates the meaning of the input sequence. This representation is then commonly utilized as input for other downstream models, such as those designed for sentence classification tasks.

Sequence Encoding Models

In Natural Language Processing, sequence generation models are designed to produce a sequence of tokens based on a given context. They are applied to various language generation tasks, including question answering and machine translation. The nature of the 'context' they use is application-dependent; for instance, in language modeling, it consists of the preceding tokens, while in machine translation, it is the source-language sequence.

Sequence Generation Models

A key architectural distinction exists between sequence encoding and generation models based on how they are applied. Generation models are typically employed as standalone systems for tasks like question answering and machine translation, operating without additional modules, which simplifies their fine-tuning. In contrast, encoding models generally serve as components within larger architectures, where their primary role is to provide input representations for other downstream models.

Architectural Differences Between Sequence Encoding and Generation Models

Diverse NLP problems can be unified under a general sequence model structure, represented by the function $$\mathbf{o} = g(x_0, x_1, ..., x_m; \theta)$$. In this formula, $$x_0, x_1, ..., x_m$$ is the input token sequence, where $$x_0$$ is a special start-of-sequence symbol (such as $$\langle s \rangle$$ or $$\mathbf{[\mathrm{CLS}]}$$). The function $$g(\cdot; \theta)$$ (also written as $$g_{\theta}(\cdot)$$) is a neural network defined by parameters $$\theta$$, and $$\mathbf{o}$$ is the model's output. A common shorthand for the output is $$\mathbf{o} = g_{\theta}(x_0, x_1, ..., x_m)$$.

General Formulation of a Sequence Model

A large language model is pre-trained on a vast text corpus. Its training objective is to take a sentence, randomly mask 15% of the words, and then predict *only* the original masked words by looking at all the surrounding unmasked words (both to the left and right). Which statement best analyzes the primary goal of this specific pre-training approach?

A research team is developing a model intended to be highly effective at summarizing long legal documents. They decide on a pre-training strategy where the model is given the first 90% of a document and is trained to produce the final 10% as its output. Analyze this pre-training strategy and determine which of the two primary problem types it represents. Justify your answer based on the model's objective.

Learn Before

Related