Input Representation in a Transformer Layer

A computational model is processing a sequence of three items (at positions 0, 1, and 2). The model's internal state at its 5th processing stage is represented by the collection of vectors $\mathbf{S}^5 = [\mathbf{v}_0^5, \mathbf{v}_1^5, \mathbf{v}_2^5]$. What does the single vector $\mathbf{v}_1^5$ represent in this context?

Representing a Layer's State for a Sequence

Applying Layer State Representation