Learn Before
Kerple
Kerple is a method introduced by Chi et al. (2022), often discussed in the context of positional bias mechanisms for transformer models, similar to ALiBi.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Interpretation of Positional Bias as a Distance Penalty
T5 Bias for Relative Positional Embedding
Shared Learnable Bias per Offset
Heuristic-Based Relative Positional Biases
Comparison of Learned vs. Heuristic-Based Relative Positional Biases
Kerple
FIRE
Relative Position Offset Calculation
A self-attention model incorporates positional awareness by adding a bias term directly to the query-key dot product for each pair of positions
(i, j). This bias term's value depends on the relative distance betweeniandj. What is the primary implication of this approach compared to the alternative of adding positional vectors to the input token embeddings?Incorporating Positional Bias into Attention Scores
In a self-attention mechanism, the score computed between a query at position
iand a key at positionjis modified by directly adding a bias term whose value depends only on the positionsiandj. What is the primary function of this bias term within the attention calculation?
Learn After
Kerple Positional Bias Formula
Kerple Logarithmic Bias Formula
Sandwich Method (Chi et al., 2023)
Formula for Relative Position Scaled by Sinusoidal Wavelength
A transformer model incorporates a positional bias mechanism where a penalty is applied to the attention score between a query and a key. This penalty grows larger as the distance between the query's position and the key's position in the sequence increases. Given the sentence 'The quick brown fox jumps over the lazy dog', which of the following query-key pairs would receive the smallest penalty from this mechanism?
Comparing Positional Bias Functions
A self-attention mechanism is modified to include a bias term that systematically penalizes attention scores between pairs of tokens. The magnitude of this penalty increases as the distance between the tokens' positions in the sequence grows. For which of the following tasks would this modification be most likely to hinder the model's performance?