Google

In language modeling, an input sequence `x` and an output sequence `y` are often processed together as a single, combined sequence. This concatenation is formally denoted as `[x, y]`, representing the sequence of tokens from `x` followed by the tokens from `y`: $x_0...x_m y_1...y_n$. An alternative notation for this combined sequence is $seq_{x,y}$.

Notation for Concatenated Token Sequences

The notation `Pr([x, y])` represents the probability that a language model will generate the specific token sequence formed by concatenating an input sequence `x` and an output sequence `y`. This is also known as the joint probability of the sequences `x` and `y`.

Probability of a Concatenated Token Sequence

An input sequence of tokens is defined as `x = (The, cat, sat)` and a subsequent output sequence is defined as `y = (on, the, mat)`. Which of the following correctly represents the single, combined sequence denoted by `[x, y]`?

A common method for preparing data for sequence models involves concatenating input and output sequences into a single string, using a special separator token like `[SEP]` to distinguish between them. For instance, an input sequence `xm` and an output sequence `y1 y2 ... yn` can be formatted as `xm [SEP] y1 y2 ... yn [SEP]`. This structure allows the model to process both sequences simultaneously while understanding their distinct roles.

Using [SEP] Tokens for Sequence Concatenation

A language model is given a combined token sequence represented by the notation `[x, y]`. The full sequence is: `(Translate, to, French, :, The, cat, is, black, Le, chat, est, noir)`. If the output sequence `y` is `(Le, chat, est, noir)`, what is the corresponding input sequence `x`?

Deconstructing a Concatenated Token Sequence

Given two token sequences, `x = (start, process)` and `y = (end, result)`, the concatenated sequence denoted by `[x, y]` is identical to the sequence denoted by `[y, x]`.

Learn Before

Related