Comparison of Position Offsets in Causal vs. Bidirectional Attention

Calculating a Relative Position Bias Bucket

The T5 relative position bias bucketing formula is a piecewise function, treating small and large relative position offsets differently. For small offsets, it uses a direct one-to-one mapping to a bucket. For larger offsets, it transitions to a logarithmic mapping. What is the primary rationale behind this dual-strategy design?

A key characteristic of the T5 relative position bias bucketing formula is that it maintains a consistent level of positional precision regardless of the distance between tokens. For example, the distinction it makes between relative positions 10 and 20 is just as fine-grained as the distinction it makes between relative positions 500 and 510.

Visualization of T5 Bias Bucketing