Calculating a Memory Component Summary
A model is designed to summarize its recent history into a single memory component. This summary is created by calculating the unweighted average of the last 3 key vectors and, separately, the last 3 value vectors. Given the following sequence of the four most recent key and value vectors, calculate the two vectors (one for keys, one for values) that would form the memory component at the current time step (step 4).
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Formula for Memory as a Moving Average of Keys and Values
Example of a Moving Average-based Cache
Cumulative Average of Keys and Values for Memory Component
Calculating a Memory Component Summary
When using a moving average of the last
nkey-value pairs to create a single summary vector for a memory component, what is the primary effect of significantly increasing the window sizen?Weighted Moving Average for Memory Component
A memory component in a transformer-based model is designed to create a summary by computing the simple, unweighted average of the last 10 key-value pairs. Which statement accurately describes a fundamental property of this specific summarization method?