Matching

An attention mechanism groups relative token distances (offsets) into buckets according to a specific scheme. Match each given offset to its correct bucket number based on the following rules:

  1. Offsets 0-15 are mapped one-to-one to buckets 0-15.
  2. For larger distances, buckets cover logarithmically increasing ranges:
    • Bucket 16: offsets 16-20
    • Bucket 17: offsets 21-26
    • Bucket 18: offsets 27-33
  3. This pattern continues until a final bucket, Bucket 32, which covers all offsets from 802 onwards.

0

1

Updated 2025-10-09

Contributors are:

Who are from:

Tags

Ch.2 Generative Models - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Application in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science

Related