Learn Before
Structure of a Transformer Layer's Input
A large language model is processing the sentence 'The cat sat,' which has been tokenized into three distinct parts. Consider the input being prepared for the 5th layer of this model. Based on the standard representation for a Transformer layer's input, describe the structure and components of the matrix, denoted as , that would be fed into this layer. What does each primary component of this matrix represent?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Transformer Layer Output Formula
General Formula for a Transformer Layer
Input Composition in a Prefix-Tuned Transformer Layer
A language model is processing an input sentence that has been broken down into 5 distinct tokens. The input to the first processing layer is represented as a matrix containing 5 separate vectors, one for each token. Why is it fundamentally important for the model to maintain this structure—a sequence of individual vectors—as the input to each subsequent layer, rather than, for example, averaging or concatenating them into a single vector?
Structure of a Transformer Layer's Input
When a Transformer model processes a sentence with 12 tokens, the input to the fifth layer is a single, high-dimensional vector that represents the aggregated meaning of the entire sentence as computed by the first four layers.