A model computes a memory component, Mem, using the following formula for a weighted moving average of the last n_c key (k) and value (v) vectors at a given position i:
Mem = ( (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j), (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} v_j) / (Σ_{j=1}^{n_c} β_j) )
Given a current position i=10, a context window size n_c=4, and weights β = [β_1, β_2, β_3, β_4], which of the following expressions correctly represents the calculation for the summary key vector?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A model computes a memory component,
Mem, using the following formula for a weighted moving average of the lastn_ckey (k) and value (v) vectors at a given positioni:Mem = ( (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} k_j) / (Σ_{j=1}^{n_c} β_j), (Σ_{j=i-n_c+1}^{i} β_{j-i+n_c} v_j) / (Σ_{j=1}^{n_c} β_j) )Given a current position
i=10, a context window sizen_c=4, and weightsβ = [β_1, β_2, β_3, β_4], which of the following expressions correctly represents the calculation for the summary key vector?Configuring Memory Component Weights
Calculating the Memory Summary Vector