Learn Before
Set of Superscript-Indexed Vectors
The notation represents a set of vectors. Each vector, denoted by , is further identified by a superscript index enclosed in square brackets, such as or . This indexing scheme is commonly used to represent a collection of vectors associated with a single element across multiple parallel contexts, for instance, the query vectors for the -th token across attention heads.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Set of Indexed Key-Value Pairs
Set of Superscript-Indexed Vectors
Set of Key-Value Pairs
Function of a Sequence of Overlined Variables
Function of a Sequence of Averaged Vectors
Vector Slice Notation for a Sequence Window ()
Set of Sequential Vectors Notation
Vector Sequence Window Notation
Consider an autoregressive model generating a sequence of tokens one by one. At each step
i, the model calculates attention using the query from the current token and the keys and values from all tokens generated so far (from position 1 toi). To optimize this process, the model maintains a growing set of all previously computed key and value vectors. What is the primary computational advantage of this strategy?State of an Autoregressive Cache
An autoregressive language model with
τparallel computational units (e.g., attention heads) is generating a sequence of tokens. After computing the output for the 3rd token, the model stores the key and value vectors from all tokens processed so far to use in subsequent steps. Which of the following notations correctly represents the complete set of these stored key-value pairs at this specific moment?
Learn After
A computational model analyzes the 4th element in a sequence. To do this, it generates 12 distinct 'probe' vectors for this single element, one for each of its 12 parallel processing modules. Which of the following notations correctly represents the complete set of all 'probe' vectors for the 4th element?
True or False: In the notation { \mathbf{q}{i}^{[1]}, \dots, \mathbf{q}{i}^{[ au]} }, the subscript
irepresents one of\tauparallel contexts, while the superscript (e.g.,[1]) refers to a specific element in a sequence.