Learn Before
Evaluating a Modification to the Linear Attention Formula
A researcher is working with a memory-efficient attention mechanism where the output for the i-th token is calculated as: In this formula, is the processed query, is an aggregation of past key-value products, and is an aggregation of past processed keys. The researcher proposes removing the denominator () to simplify the computation. Evaluate this proposal. What essential function, typically performed by a different operation in standard attention mechanisms, would be lost? What would be the likely impact on the model's output stability and overall performance?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
In a memory-efficient attention mechanism, the output for a token at position
iis calculated using the formula:Output = (q'_i * μ_i) / (q'_i * ν_i). In this formula,q'_iis the token's processed query, whileμ_iandν_iare aggregations of historical information from all tokens up to and including positioni. Specifically,μ_iaggregates past key-value products, andν_iaggregates past keys. What is the primary function of the denominator,q'_i * ν_i?Efficiency of Aggregated State in Attention
Evaluating a Modification to the Linear Attention Formula
In the formula for calculating a linear attention output,
Output = (q'_i * μ_i) / (q'_i * ν_i), whereq'_iis the transformed query,μ_iis the accumulated key-value state, andν_iis the accumulated key state, what is the primary role of the denominator termq'_i * ν_i?Calculating a Linear Attention Output Vector
Recurrent Computation of and in Linear Attention