A model's memory component is calculated as the unweighted moving average of the last n_c key and value vectors. Given the following sequence of 2-dimensional key (k) and value (v) vectors at four consecutive time steps, and a context window size n_c = 3, what is the memory component (average_key, average_value) at the fourth time step (i=4)?

k_1 = [1, 2], v_1 = [10, 11] k_2 = [3, 4], v_2 = [12, 13] k_3 = [5, 6], v_3 = [14, 15] k_4 = [7, 8], v_4 = [16, 17]

Impact of Context Window Size on Memory

Consider the formula for a memory component calculated as an unweighted moving average of the last n_c key and value vectors: Mem = ( (Σ k_j) / n_c, (Σ v_j) / n_c ). If the context window size n_c is increased, the influence of any single key-value pair (k_j, v_j) within that window on the final memory component will also increase.