Learn Before
Example of State Definition for Next-Token Prediction
As an example of defining a state for next-token prediction in a language model, the state at time step 't' can be formally represented as the pair (x, yt), where 'x' is the initial input sequence and 'yt' is the sequence of tokens generated up to that point.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Example of State Definition for Next-Token Prediction
A language model is given the initial text 'The sun is shining and the sky is'. The model then generates the word 'blue'. At this point, before it attempts to generate the next word, what sequence of tokens represents the model's current 'state' that it will use as context?
The Role of State in Language Models
When a language model generates a new token, the 'state' it uses for the next prediction is updated to include only the token it just produced, discarding all previously seen tokens.
Learn After
A language model is provided with the initial input sequence 'x' = "The sun is shining and the". The model then generates a sequence of three tokens, 'yt' = "sky is blue". According to the formal state definition where the state is a pair of the initial input and the generated sequence, what is the correct representation of the model's state at this point?
Analyzing Language Model State Evolution
State Evolution in Token Generation