Impact of Context Window Size on Memory
A memory component is defined by the formula below, which calculates the unweighted moving average of the last n_c key and value vectors. Analyze the primary trade-off when selecting the value for n_c. Specifically, contrast the characteristics of the memory component when n_c is very small versus when it is very large.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A model's memory component is calculated as the unweighted moving average of the last
n_ckey and value vectors. Given the following sequence of 2-dimensional key (k) and value (v) vectors at four consecutive time steps, and a context window sizen_c = 3, what is the memory component(average_key, average_value)at the fourth time step (i=4)?k_1 = [1, 2],v_1 = [10, 11]k_2 = [3, 4],v_2 = [12, 13]k_3 = [5, 6],v_3 = [14, 15]k_4 = [7, 8],v_4 = [16, 17]Impact of Context Window Size on Memory
Consider the formula for a memory component calculated as an unweighted moving average of the last
n_ckey and value vectors:Mem = ( (Σ k_j) / n_c, (Σ v_j) / n_c ). If the context window sizen_cis increased, the influence of any single key-value pair(k_j, v_j)within that window on the final memory component will also increase.