Learn Before
Calculating Linear Relative Position Biases
In a causal attention mechanism, a linear relative position bias is calculated using the formula bias = -β * (i - j), where i is the query position, j is the key position, and β is a positive scalar. If the query is at position 6 (i=6) and it can only attend to keys at its own position and all previous positions, what are the bias values for the keys at positions 6, 5, 4, and 3? Express your answer as a sequence of four values in terms of β, ordered from position 6 down to 3.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a text-processing model, a bias term is added to the attention scores between a 'query' word and a 'key' word before the final attention weights are computed. This bias is calculated as
-β * d, wheredis the distance (number of words) between the query and the key, andβis a fixed positive number. What is the primary effect of this biasing scheme on the model's behavior?Calculating Linear Relative Position Biases
An attention mechanism uses a linear relative position bias to penalize distant key-value pairs. In a causal setting, a query at a given position attends to itself and all previous positions up to a certain maximum distance. Match each maximum relative distance to the corresponding set of bias values that would be applied, where β is a scalar.