Calculating the Memory Summary Vector
A memory component is calculated as a weighted moving average of key vectors (k) and value vectors (v). The formula for the summary key vector is given by:
Summary Key = (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j)
Where i is the current position, n_c is the context window size, and β is a vector of weights.
Given the following:
- Current position
i = 5 - Context window size
n_c = 3 - Weights
β = [β_1, β_2, β_3] = [0.1, 0.3, 0.6] - Key vectors:
k_3 = [2, 4]k_4 = [1, 3]k_5 = [5, 0]
Calculate the summary key vector. Provide your answer as a vector (e.g., [x, y]).
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A model computes a memory component,
Mem, using the following formula for a weighted moving average of the lastn_ckey (k) and value (v) vectors at a given positioni:Mem = ( (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j), (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} v_j) / (Σ_{j=1}^{n_c} β_j) )Given a current position
i=10, a context window sizen_c=4, and weightsβ = [β_1, β_2, β_3, β_4], which of the following expressions correctly represents the calculation for the summary key vector?Configuring Memory Component Weights
Calculating the Memory Summary Vector