Learn Before
Analyzing Unexpected Attention Output
Imagine an attention mechanism where, for a specific step, the calculated attention weights for three input items are [0.9, 0.05, 0.05]. The first item is the most relevant. Despite this strong focus, the final output vector is found to be almost entirely composed of the information from the second input item (the one with a weight of 0.05). What is the most likely reason for this discrepancy, considering how the final output is constructed?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a simplified attention mechanism processing an input sequence, the attention scores for a particular output are calculated as [0.1, 0.8, 0.1] for the three input items respectively. If the information-carrying vector for the second input item (the one with the 0.8 score) was replaced with a zero vector (a vector containing only zeros), what would be the most direct consequence for the output of the attention layer?
Analyzing Unexpected Attention Output
Calculating Attention Output