Self-Attention
Self-attention, also referred to as intra-attention, is an attention mechanism that relates different positions within a single sequence to compute a representation of that sequence.
0
1
Tags
Prep Sessions
Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture Fundamentals - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Self-Attention and Query-Key-Value Mechanisms - Transformer Architecture and Large Language Model Capabilities @ University of Michigan - Ann Arbor
Learn After
Which of the following is another name for self-attention?
True or False: Self-attention operates by connecting positions across two or more separate input sequences.
What does self-attention compute for an input sequence?
Consider how self-attention processes an ambiguous word in a sentence, such as 'bank' in the phrase 'river bank'. Explain how the mechanism uses the sentence to determine the contextual meaning of that word.