Learn Before
A language model is given the initial text 'The sun is shining and the sky is'. The model then generates the word 'blue'. At this point, before it attempts to generate the next word, what sequence of tokens represents the model's current 'state' that it will use as context?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of State Definition for Next-Token Prediction
A language model is given the initial text 'The sun is shining and the sky is'. The model then generates the word 'blue'. At this point, before it attempts to generate the next word, what sequence of tokens represents the model's current 'state' that it will use as context?
The Role of State in Language Models
When a language model generates a new token, the 'state' it uses for the next prediction is updated to include only the token it just produced, discarding all previously seen tokens.