Learn Before
Impact of Query-Key Vector Similarity
In an attention mechanism, a Query vector is compared against a set of Key vectors to calculate attention scores, which determine the relevance of different parts of an input. If a particular Query vector is mathematically identical to one of the Key vectors in the input sequence, what is the expected effect on the attention score associated with that Key vector, and what is the underlying reason for this effect?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is tasked with translating the English sentence 'The quick brown fox jumps over the lazy dog' into another language. The model has already generated the partial translation 'The quick brown fox...' and is now in the process of generating the word that corresponds to 'jumps'. At this specific moment, what does the model's Query vector represent?
Role of the Query Vector in Text Summarization
Impact of Query-Key Vector Similarity