Impact of Masking on Attention Weight Distribution
A model is processing a sequence of three input tokens. For the query vector at position 2, the un-normalized attention scores calculated with respect to the key vectors at positions 0, 1, and 2 are [2.5, 1.0, 4.0]. Now, imagine a mechanism is applied that prevents the query at position 2 from attending to itself. This is achieved by adding a very large negative number (e.g., -10,000) to the score for position 2 before the normalization step that converts scores into weights. Analyze and explain the resulting effect on the final, normalized attention weights for the query at position 2.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Scaled Dot-Product Attention
Causal Self-Attention in Autoregressive Decoders
A model is processing a sequence of three tokens. For the query at position 2, the un-normalized attention scores with respect to the keys at positions 0, 1, and 2 are calculated as [1.0, 2.0, 3.0] respectively. What is the final attention weight that the token at position 2 will assign to the token at position 1?
Attention Output as a Weighted Sum of Values
Impact of Masking on Attention Weight Distribution
True or False: In a self-attention mechanism, if you add the same constant value to all un-normalized attention scores corresponding to a single query vector, the final normalized attention weights for that query will change.
Attention Weight Formula ()