Formula for Memory as a Weighted Moving Average of Keys and Values

Increasing Coefficients as a Heuristic for Weighted Moving Average

A language model's memory component creates a summary vector of past information using a weighted moving average. The weights are determined by a heuristic that assigns significantly higher importance to more recent information. For a task like summarizing a long, complex article, what is the most probable impact of this specific weighting scheme on the model's output?

Learned vs. Heuristic Weights for Memory Summarization

Configuring Memory for Narrative Coherence