An engineer is training a model on extremely long text sequences and notices that the training process is unstable. Upon inspection, they find that the attention mechanism frequently assigns nearly all of its weight to a single, specific token, ignoring the rest of the context. This extreme concentration of attention scores is causing gradients to become erratic. Which of the following strategies directly addresses this problem by encouraging a less concentrated, more stable distribution of atte

A machine learning engineer is training a model on very long documents and observes that the attention mechanism is behaving unstably: for some parts of the text, the attention weights are extremely concentrated on a single word, while for others, they are so spread out they are effectively zero. After implementing a change, the engineer observes that the attention weights are now more smoothly distributed, leading to more consistent model performance. Which of the following changes best explain

Evaluating Attention Mechanism Stability