Learn Before
Consequences of Non-Unique Identifiers in Attention
In a system designed to focus on relevant parts of an input sentence, each word is assigned a 'label' vector that acts as its identifier. A 'context' vector, representing the current point of focus, is then compared against every word's 'label' vector to calculate a relevance score. If, due to a system error, every word in the sentence 'The quick brown fox jumps' was assigned an identical 'label' vector, what would be the consequence for the calculated relevance scores? Explain your reasoning.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is processing the sentence: 'The delivery driver carefully parked the large van.' To understand the context of the word 'parked', the model generates a 'focus' vector for it. This 'focus' vector is then compared against a 'label' vector for every word in the sentence to calculate relevance scores. What is the primary function of the 'label' vector associated with the word 'van' in this process?
Diagnosing Attention Mechanism Failure
Consequences of Non-Unique Identifiers in Attention