1Cademy - Formula for Logarithmic Bucketing in T5 Bias

Learn Before

Logarithmic Bucketing for Larger T5 Offsets

Formula

Formula for Logarithmic Bucketing in T5 Bias

In the T5 bias mechanism, the bucket number $b(i-j)$ for a large offset, where $i - j \ge \frac{n_b+1}{2}$ , is calculated using a logarithmic scale. The formula is defined as:

$b(i - j) = \frac{n_b+1}{2} + \left\lfloor \frac{\log(i - j) - \log(\frac{n_b+1}{2})}{\log(\mathrm{dist}_{\mathrm{max}}) - \log(\frac{n_b+1}{2})} \cdot \frac{n_b+1}{2} \right\rfloor$

This equation maps the offset to a bucket index by normalizing its logarithmic position relative to a maximum distance, $\mathrm{dist}_{\mathrm{max}}$ , and scaling it to the available number of logarithmic buckets.