Google

A straightforward method for creating summary vectors for the memory component ($$\mathrm{Mem}$$) is to compute the moving average of past keys and values. For instance, $$\mathrm{Mem}$$ can be constructed from the unweighted moving average of the last $$n_c$$ key and value vectors, providing a simple yet effective way to summarize history.

Moving Average of Keys and Values for Memory Component

The memory component, $$\mathrm{Mem}$$, can be defined as a pair of summary vectors calculated by taking the unweighted moving average of the last $$n_c$$ key and value vectors. This is formally expressed as: $$\mathrm{Mem} = \Big(\frac{\sum_{j=i - n_c + 1}^{i} \mathbf{k}_{j}}{n_c} , \frac{\sum_{j=i - n_c + 1}^{i} \mathbf{v}_{j}}{n_c} \Big)$$

Formula for Memory as a Moving Average of Keys and Values

A moving average-based cache compresses a window of recent key-value pairs into a single summary pair. As illustrated, the four most recent key vectors ($\mathbf{k}_{i-3}$ to $\mathbf{k}_{i}$) and value vectors ($\mathbf{v}_{i-3}$ to $\mathbf{v}_{i}$) are averaged independently. This process creates a memory component consisting of one summary key and one summary value, effectively reducing the memory size from four pairs to one (a size of 1x2). The specific calculations are:
$$ \text{Memory Key} = \frac{\mathbf{k}_{i-3}+\mathbf{k}_{i-2}+\mathbf{k}_{i-1}+\mathbf{k}_{i}}{4} $$
$$ \text{Memory Value} = \frac{\mathbf{v}_{i-3}+\mathbf{v}_{i-2}+\mathbf{v}_{i-1}+\mathbf{v}_{i}}{4} $$

Example of a Moving Average-based Cache

The moving average approach for creating memory summary vectors can be extended to a cumulative average of the keys and values. Instead of averaging over a fixed-size window of recent key-value pairs, this method extends the moving average to include all positions from the beginning of the sequence up to the current position $$i$$. This results in a summary that incorporates the entire history of the sequence at each step.

Cumulative Average of Keys and Values for Memory Component

A model is designed to summarize its recent history into a single memory component. This summary is created by calculating the unweighted average of the last 3 key vectors and, separately, the last 3 value vectors. Given the following sequence of the four most recent key and value vectors, calculate the two vectors (one for keys, one for values) that would form the memory component at the current time step (step 4).

Calculating a Memory Component Summary

When using a moving average of the last `n` key-value pairs to create a single summary vector for a memory component, what is the primary effect of significantly increasing the window size `n`?

To give varying levels of importance to past information, a weighted moving average can be used to create summary vectors for the memory component ($$\mathrm{Mem}$$). This method applies different weights, or coefficients ($$\beta_1, \dots, \beta_{n_c}$$), to the key and value vectors within the attention window. The specific values for these coefficients can be either learned as model parameters or determined via heuristics.

Weighted Moving Average for Memory Component

A memory component in a transformer-based model is designed to create a summary by computing the simple, unweighted average of the last 10 key-value pairs. Which statement accurately describes a fundamental property of this specific summarization method?

Learn Before

Related