In the ALiBi framework, the positional bias term is defined as the offset between the query and key positions, which is then multiplied by a negative scalar. This method effectively imposes a linear penalty on the attention score that increases with the distance between tokens.

ALiBi Bias Term Definition

Attention with Linear Biases, or ALiBi, is a specific implementation of a heuristic-based relative positional bias, as introduced by Press et al. (2022). This method avoids learning positional parameters by instead calculating a bias for the query-key product directly from the distance between tokens.

Google

An alternative to learned relative positional embeddings involves using fixed bias values determined by heuristics. This method does not require training on a specific dataset, allowing the biases to be applied directly to any sequence once they are established.

Learn Before

Related

Learn After