In a memory-efficient attention mechanism, the output for a token at position i is calculated using the formula: Output = (q'_i * μ_i) / (q'_i * ν_i). In this formula, q'_i is the token's processed query, while μ_i and ν_i are aggregations of historical information from all tokens up to and including position i. Specifically, μ_i aggregates past key-value products, and ν_i aggregates past keys. What is the primary function of the denominator, q'_i * ν_i?

Efficiency of Aggregated State in Attention

Evaluating a Modification to the Linear Attention Formula

In the formula for calculating a linear attention output, Output = (q'_i * μ_i) / (q'_i * ν_i), where q'_i is the transformed query, μ_i is the accumulated key-value state, and ν_i is the accumulated key state, what is the primary role of the denominator term q'_i * ν_i?

Calculating a Linear Attention Output Vector

Recurrent Computation of $\mu_i$ and $\nu_i$ in Linear Attention