A language model is given an incomplete sentence and generates a completion. Based on the formal structure of a model's output, how would you deconstruct the generated text into its fundamental components? Describe the overall structure and identify what the first few components would likely be.

Google

The output generated by a Large Language Model, often referred to as the response, is structured as a token sequence. This sequence is represented by $y$ and consists of $n$ tokens, denoted as $y_1...y_n$.

Output Token Sequence in LLMs

The inference process in Large Language Models (LLMs) is formally defined as finding the most probable output sequence based on a given user context. Let $$\mathbf{x}$$ denote the input token sequence (conceptually equivalent to a 'prompt'), which comprises $$m+1$$ tokens denoted by $$x_0...x_m$$, where $$x_0$$ is the start symbol $$\langle \mathrm{SOS} \rangle$$. Let $$\mathbf{y}$$ denote the subsequent output token sequence (the response), comprising $$n$$ tokens denoted by $$y_1...y_n$$. The output tokens preceding position $$i$$ are denoted as $$\mathbf{y}_{<i} = y_1...y_{i-1}$$. The primary goal of LLM inference is to maximize the conditional probability $$\Pr(\mathbf{y}|\mathbf{x})$$, evaluating the context $$\mathbf{x}$$ to determine the most likely sequence $$\mathbf{y}$$. Furthermore, the input and output can be concatenated into a single sequence $$[\mathbf{x},\mathbf{y}] = x_0 ... x_m y_1 ... y_n$$ (sometimes represented as $$\mathrm{seq}_{\mathbf{x},\mathbf{y}}$$) to compute joint log-probabilities in decoder-only models.

Formal Definition of LLM Inference

In autoregressive language models, the notation $y_{<i}$ is commonly used to represent the subsequence of output tokens that precede position $i$. This sequence, consisting of $y_1...y_{i-1}$, provides the context for predicting the token $y_i$. An alternative notation for this same subsequence is `yi`.

Notation for Preceding Output Subsequence

Deconstructing a Model's Generated Text

A language model generates the sentence: 'The capital of France is Paris.' Assuming each word and the final punctuation mark is treated as a separate token, represent this output as a token sequence $y$ consisting of $n$ tokens ($y_1...y_n$). Your answer should clearly state the value of $n$ and list each individual token from $y_1$ to $y_n$.

Representing Model Output as a Token Sequence

A Large Language Model generates the sentence: 'AI is transforming our world.' How is this output fundamentally structured by the model before being presented to the user?

Although the input and output tokens of a Large Language Model can technically be viewed as sub-sequences of a single, continuous sequence, it is common practice to employ separate variables—typically $$\mathbf{x}$$ for the input and $$\mathbf{y}$$ for the output. Adopting this distinct notation helps to clearly separate the given context from the generated text, resulting in mathematical formulations that closely resemble those used in other natural language processing text generation models, such as neural machine translation.

Learn Before

Related