Learn Before
Increasing Coefficients as a Heuristic for Weighted Moving Average
A common heuristic for setting the coefficients in a weighted moving average is to use values that increase for more recent items. This gives larger weight to positions that are closer to the current position . This can be represented by a sequence of coefficients, , that are strictly increasing, as shown by the inequality: In this setup, would be the coefficient for the oldest item in the moving average window, and would be the coefficient for the most recent item.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Formula for Memory as a Weighted Moving Average of Keys and Values
Increasing Coefficients as a Heuristic for Weighted Moving Average
A language model's memory component creates a summary vector of past information using a weighted moving average. The weights are determined by a heuristic that assigns significantly higher importance to more recent information. For a task like summarizing a long, complex article, what is the most probable impact of this specific weighting scheme on the model's output?
Learned vs. Heuristic Weights for Memory Summarization
Configuring Memory for Narrative Coherence
Learn After
A system is designed to compute a summary value at each time step by calculating a weighted average of the last four items in a sequence. The core design principle is that items closer to the current time step should have a greater influence on the summary than items from further in the past. If the weights are applied to the items in order from oldest to most recent, which of the following sets of coefficients best implements this principle?
Improving a Time-Sensitive Prediction System
Rationale for Increasing Weights in a Moving Average