Deconstructing an Input Vector
A language model is given the following pair of sentences, formatted as a single input sequence:
[CLS] What is the weather like? [SEP] It is sunny and warm. [SEP]
The final input vector for each word is created by summing three separate vectors. Analyze the final input vector for the word 'sunny'. Describe the three component vectors that are added together to create it, explaining the specific information each component contributes in this context.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Consider the following input prepared for a language model that processes sentence pairs:
[CLS] The sky is blue . [SEP] The grass is green . [SEP]The final vector representation for each token in this sequence is created by summing three distinct vectors: one for the token's identity, one for its position in the sequence, and one to indicate which sentence it belongs to (the first or the second).
Which of the following statements correctly analyzes the final vector representations for the two instances of the token 'The' in the sequence?
Deconstructing an Input Vector
Debugging an Input Embedding