Learn Before
Analysis of Sparse Attention Patterns
A team is optimizing a causal language model and is evaluating two different sparsity patterns for computing the attention output at token position i=20. The patterns are defined by the following index sets of non-zero attention weights. Based on these sets, which pattern implies a higher degree of sparsity (and thus greater computational efficiency), and which pattern is designed to focus more on the most recent context? Justify your reasoning for both.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A causal language model uses a sparse attention mechanism. When calculating the output for the token at position
i=10, the set of indices for the key-value pairs to be attended to is specified asG = {2, 5, 9}. Which of the following statements accurately describes the computation for the token at position 10?A causal language model is using a sparse attention mechanism to compute the output for the token at position
i = 8. The setGdefines the indices of the key-value pairs that the current token will attend to. Which of the following options represents an invalid setGfor this computation?Analysis of Sparse Attention Patterns
Sparsity Level and the Size of Index Set