Analysis of Attention Head Architectures
Imagine two different designs for a model's attention component. In 'Design 1', each attention head calculates its output using its own unique Query, Key, and Value vectors. In 'Design 2', each attention head still uses its own unique Query vector, but all heads must use a single, shared set of Key and Value vectors. Based on this information, analyze the fundamental difference in the inputs provided to a single attention head in 'Design 2' compared to 'Design 1'. What is the primary structural consequence of adopting 'Design 2'?
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Analysis of Attention Head Architectures
An engineer is analyzing the computational architecture of a large language model. They observe the following formula being used to calculate the output for an individual attention head
jat a specific stepi:head_j = Attention(q_i^[j], K_<=i, V_<=i)Based only on the components of this formula, what is the most accurate conclusion the engineer can draw about the relationship between the different attention heads in this layer?
In a Multi-Query Attention (MQA) layer, all attention heads share the same Key and Value matrices. The formula for the output of a single, specific head
jat stepiis given as:head_j = Att_qkv(______, K_<=i, V_<=i). What component correctly fills the blank to represent the unique input for this specific head?