Positional Encoding
Unlike Recurrent Neural Networks (RNNs) that process tokens sequentially one-by-one, self-attention ditches sequential operations in favor of parallel computation and does not naturally preserve the order of the input sequence. To address this order-insensitivity, the dominant approach is to represent the sequence order as an additional input associated with each token, called a positional encoding. These encodings can be either learned during training or fixed a priori.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
D2L
Dive into Deep Learning @ D2L
Prep Sessions
Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Ch.1 Transformer Architecture and Components - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Sinusoidal and Learned Positional Encodings - Foundational Deep Learning Architectures: Transformers and Residual Networks @ University of Michigan - Ann Arbor
Related
Standard Transformer Encoding Procedure
Key Hyperparameters of a Transformer Encoder
Transformer Encoding of a Masked Bilingual Sentence Pair
Prefix Tuning
In a sequence-to-sequence model, the input is processed by a stack of six encoder layers that have identical structures. A proposal is made to modify this architecture so that all six encoder layers share the exact same set of weights, with the goal of reducing the total number of model parameters. Which statement best analyzes the primary consequence of this change on the model's ability to process information?
A sentence is fed into the encoder side of a Transformer model. Arrange the following steps in the correct sequence to describe how the initial input is processed by the stack of encoders.
Improving a Transformer's Contextual Understanding
Positional Encoding
Transformer Encoder Sublayers
Sinusoidal versus Learned Positional Embeddings
Positional Encoding
Sinusoidal Positional Encoding
Learn After
Self-Attention layer understanding - Step 5 - Adding the time
Input Embedding with Positional Encoding
Learnable Absolute Positional Embeddings
Initial Input Representation for Transformer Layers
Comparison of Arbitrary Order Prediction and Masked Language Modeling
Model Architecture Design Choice
Positional Encoding Vector
How does the computation method of self-attention differ from that of Recurrent Neural Networks (RNNs)?
What are the two ways that positional encodings can be established to provide sequence order information?