In the context of a language model generating text one token at a time, explain what constitutes the model's 'state' at any given step and describe the primary purpose of this state in the generation process.

Google

In language modeling, a state at a specific time step, denoted as $$s$$, is defined as the sequence of tokens observed up to that point. This sequence serves as the context the model utilizes to predict the subsequent token. For instance, when predicting the next token at time step $$t$$, the state can be mathematically defined as $$(\mathbf{x}, \mathbf{y}_{< t})$$, where $$\mathbf{x}$$ represents the initial input and $$\mathbf{y}_{< t}$$ represents the generated tokens so far.

State in the Context of LLMs

As an example of defining a state for next-token prediction in a language model, the state at time step 't' can be formally represented as the pair (x, yt), where 'x' is the initial input sequence and 'yt' is the sequence of tokens generated up to that point.

Example of State Definition for Next-Token Prediction

A language model is given the initial text 'The sun is shining and the sky is'. The model then generates the word 'blue'. At this point, before it attempts to generate the next word, what sequence of tokens represents the model's current 'state' that it will use as context?

The Role of State in Language Models

When a language model generates a new token, the 'state' it uses for the *next* prediction is updated to include only the token it just produced, discarding all previously seen tokens.

Learn Before

Related