Input Vector Creation in Autoregressive Generation
An autoregressive language model is generating text. After producing a token at a specific position in the sequence, the model must create a single vector to represent that token before feeding it into its processing layers. Describe the two distinct types of information that are typically combined to form this initial input vector.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Comprehension in Revised Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An autoregressive language model is generating text one token at a time. It has just produced the token 'blue' as the fourth token in the sequence 'The sky is blue'. To determine the fifth token, the model must first create an input representation for the token 'blue' at position 4. How is this initial representation for 'blue' typically constructed before it is fed into the model's processing layers?
Input Vector Creation in Autoregressive Generation
Input Vector Construction During Text Generation