Based on the case study, analyze the connection between the model's poor performance on long-distance reorderings and the specific design of its positional bias term. Explain why this design is ill-suited for the described translation task.

Google

In self-attention with relative positional embeddings, the bias term $$ \mathrm{PE}(i, j) $$ added to the query-key product can intuitively be interpreted as a distance penalty between positions $$ i $$ and $$ j $$. To reflect that tokens further apart should generally have less influence on each other, the value of $$ \mathrm{PE}(i, j) $$ decreases as the token at position $$ i $$ moves further away from the token at position $$ j $$.

Interpretation of Positional Bias as a Distance Penalty

In a self-attention mechanism, a bias term is added to the score calculated between a query at position `i` and a key at position `j`. Consider a scenario where this bias term is designed to be a large negative value when the distance `|i - j|` is large, and it approaches zero as the distance gets smaller. How would this specific design influence the model's behavior?

Diagnosing Model Behavior via Positional Bias

Imagine you are designing a self-attention model for a task that requires focusing primarily on the relationships between words that are very close to each other in a sentence. How would you structure the positional bias term, which is added to the attention score between position `i` and position `j`, to encourage this behavior? Describe the relationship between the bias value and the distance between the two positions (`|i - j|`).

Learn Before

Related