Learn Before
Encoder Pre-training Output Architecture
To overcome the lack of direct supervision signals in encoder pre-training, a typical approach is to combine the encoder with output layers that generate easier-to-obtain supervision signals. For instance, adding a Softmax layer on top of a Transformer encoder creates an architecture identical to a decoder-based language model, allowing the system to output a sequence of probability distributions for training.
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A model processes the input sentence 'The cat sat.' which is broken down into a sequence of 4 tokens: ['The', 'cat', 'sat', '.']. If this model functions as a sequence encoder, what is the most accurate description of the output it generates?
Model Output for a Token-Level Task
A sequence encoder processes an input sequence of 10 tokens and produces a single, fixed-size vector that represents the entire sequence's meaning.
Probabilistic Model for Text Classification using an Encoder-Classifier Architecture
Challenge of Encoder Pre-training Evaluation
Encoder Pre-training Output Architecture