Calculating a Linear Attention Output Vector
In a linear attention mechanism, at a specific timestep i, you are given the following:
- Transformed query vector:
q'_i = [2, 1] - Accumulated key-value state:
μ_i = [[10, 5], [4, 8]] - Accumulated key state:
ν_i = [6, 3]
Using the formula Output = (q'_i * μ_i) / (q'_i * ν_i), calculate the final output vector. Provide the calculated values for the numerator and the denominator before giving the final result.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In the formula for calculating a linear attention output,
Output = (q'_i * μ_i) / (q'_i * ν_i), whereq'_iis the transformed query,μ_iis the accumulated key-value state, andν_iis the accumulated key state, what is the primary role of the denominator termq'_i * ν_i?Calculating a Linear Attention Output Vector
In a memory-efficient attention mechanism, the output for a token at position
iis calculated using the formula:Output = (q'_i * μ_i) / (q'_i * ν_i). In this formula,q'_iis the token's processed query, whileμ_iandν_iare aggregations of historical information from all tokens up to and including positioni. Specifically,μ_iaggregates past key-value products, andν_iaggregates past keys. What is the primary function of the denominator,q'_i * ν_i?Efficiency of Aggregated State in Attention
Evaluating a Modification to the Linear Attention Formula
In the formula for calculating a linear attention output,
Output = (q'_i * μ_i) / (q'_i * ν_i), whereq'_iis the transformed query,μ_iis the accumulated key-value state, andν_iis the accumulated key state, what is the primary role of the denominator termq'_i * ν_i?Calculating a Linear Attention Output Vector
Recurrent Computation of and in Linear Attention