Learn Before
Kerple Positional Bias Formula
The Kerple method for positional bias in transformer models utilizes a power-law function, as indicated by the term '(power)' in its representation. The bias between a query at position and a key at position is calculated with the formula: In this formula, and are hyperparameters that define the scale and exponent of the power-law penalty, respectively.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Kerple Positional Bias Formula
Kerple Logarithmic Bias Formula
Sandwich Method (Chi et al., 2023)
Formula for Relative Position Scaled by Sinusoidal Wavelength
A transformer model incorporates a positional bias mechanism where a penalty is applied to the attention score between a query and a key. This penalty grows larger as the distance between the query's position and the key's position in the sequence increases. Given the sentence 'The quick brown fox jumps over the lazy dog', which of the following query-key pairs would receive the smallest penalty from this mechanism?
Comparing Positional Bias Functions
A self-attention mechanism is modified to include a bias term that systematically penalizes attention scores between pairs of tokens. The magnitude of this penalty increases as the distance between the tokens' positions in the sequence grows. For which of the following tasks would this modification be most likely to hinder the model's performance?