These representations focus on positional relationships between tokens instead of positions of individual tokens. This works because pairwise positional relationships between input elements (direction and distance) can often be more beneficial than positions of element.

Relative Positional Representations

Implicit representations encode positional information in word embeddings by generalizing embedding to continuous functions over positions rather than explicitly adding positional embeddings

Implicit Positional Representations

Hybrid approaches have been proposed to make use of both absolute and relative positional encodings

Other representations of positional information in transformers

A straightforward approach for encoding positional context is to treat the positional embedding for each location, $$\mathrm{PE}(i)$$, as a set of learnable parameters or variables. These parameters are optimized and trained alongside the main model parameters. This technique enables the model to develop a unique vector representation for every position, thereby allowing it to differentiate between tokens based on their positions within a sequence.

Learnable Absolute Positional Embeddings

Similar to sinusoidal embeddings, rotary positional embeddings (RoPE) utilize fixed, hard-coded values to represent positions. However, instead of adding positional vectors to token embeddings, RoPE models positional context by rotating the token embeddings in a complex vector space. This results in a multiplicative integration of positional information, distinguishing it from the additive approach common in other methods.

Rotary Positional Embeddings

These are mechanisms that inject positional information into Transformers. Positional encodings are important within the transformer architecture because the ordering of words matters when modeling sequences of text.


Types of Positional Representations:
- absolute positional representations
- relative position representations
- implicit representations
- other representations

Carleton College

Variant with an augmentation to a specific module of the transformer model

Module-Level variant

Lin, Tianyang & Wang, Yuxin & Liu, Xiangyang & Qiu, Xipeng. (2021). A Survey of Transformers. 

A Survey of Transformers (Lin et. al, 2021)

Improvements to the attention module can be categorized into these directions:

1. Sparse Attention
2. Linearized Attention
3. Prototype and Memory Compression
4. Low-rank Self-Attention
5. Attention with Prior
6. Improved Multi-Head Attention

Attention-level improvements of Transformers

Positional Representations of Transformers

Here are a few modifications proposed to improve the feed-forward network (FFN) of the transformer:


- changes to the activation function
- Adapting the FFN for Larger Capacity
- Dropping FFN Layers

Improvements to the FFN of a transformer

Layer normalization (LN) is a widely-used architectural modification that is critical for stabilizing the training of deep networks like Transformers. It operates by normalizing the inputs across all features for each training example independently. The specific mathematical function used for layer normalization is central to its application. Key areas of research and improvement for LN in transformers include its placement within the architecture, the development of effective substitutes, and the creation of normalization-free models.

Learn Before

Related

Learn After