Self-Attention
Self-attention, also referred to as intra-attention, is an attention mechanism that relates different positions within a single sequence to compute a representation of that sequence.
0
1
References
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture Fundamentals - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Self-Attention and Query-Key-Value Mechanisms - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Learn After
Which of the following is another name for self-attention?
True or False: Self-attention operates by connecting positions across two or more separate input sequences.
What does self-attention compute for an input sequence?
Consider how self-attention processes an ambiguous word in a sentence, such as 'bank' in the phrase 'river bank'. Explain how the mechanism uses the sentence to determine the contextual meaning of that word.
Learn After
Which of the following is another name for self-attention?
True or False: Self-attention operates by connecting positions across two or more separate input sequences.
What does self-attention compute for an input sequence?
Consider how self-attention processes an ambiguous word in a sentence, such as 'bank' in the phrase 'river bank'. Explain how the mechanism uses the sentence to determine the contextual meaning of that word.