Positional Encoding Strategy for a Long-Context Model

A research team is pre-training a new large language model with the goal of processing documents much longer than those typically seen in standard benchmarks. They observe that while the model performs well on shorter texts, its performance sharply degrades on sequences longer than the maximum length used during its initial training phase. The model seems unable to understand the relationships between tokens that are far apart in these extended contexts. Which of the following is the most probable cause of this issue?

Comparing Positional Embedding Strategies for Long-Context Pre-training