Analyzing a Positional Bias Mechanism

A researcher is implementing a positional bias mechanism for a multi-layer transformer model, as introduced by Chi et al. in 2023. The goal is to influence the attention scores based on the relative positions of tokens. Given the specific design of this method, which of the following implementation strategies is correct?

The positional bias mechanism introduced by Chi et al. in 2023 applies its bias within every self-attention layer of a transformer model to ensure consistent spatial awareness is maintained throughout the entire network stack.

Sandwich Positional Bias Formula