Learn Before
Training Decoder-Only Language Models with Cross-Entropy Loss
Training a decoder-only language model, represented as , involves optimizing its parameters by minimizing a loss function across a collection of token sequences. At each token position , the difference between the model's predicted probability distribution for the next token () and the true prediction or gold-standard distribution () is quantified using a loss function, denoted as . In natural language processing, the standard choice for this loss calculation is the log-scale cross-entropy loss.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Ch.4 Alignment - Foundations of Large Language Models
Related
Training Decoder-Only Language Models with Cross-Entropy Loss
Output Probability Calculation in Transformer Language Models
Global Nature of Standard Transformer LLMs
Processing Flow of Autoregressive Generation in a Decoder-Only Transformer
Initial Input Representation for Transformer Layers
Greedy Decoding in Language Models
Structure of a Transformer Block
A generative language model is designed to produce text by predicting the next token based solely on the sequence of tokens that came before it. If you were to adapt a standard Transformer decoder block for this specific auto-regressive task, which of its sub-layers would you remove, and why is this modification functionally necessary?
A language model is constructed using a stack of modified Transformer decoder blocks. Each block contains a self-attention sub-layer and a feed-forward network sub-layer, but lacks the sub-layer that would process information from a separate, secondary input sequence. This model is capable of performing a machine translation task, such as translating a German sentence into English, without any further architectural changes.
Function of Self-Attention in Auto-regressive Generation
Neural Network-Based Next-Token Probability Distribution
Learn After
Total Loss Calculation for a Token Sequence
An auto-regressive language model is being trained on the text sequence: 'The quick brown fox jumps'. At the training step where the model has processed the input 'The quick brown fox', what two quantities are compared by the cross-entropy loss function to calculate the error signal for updating the model's parameters?
Language Model Training Step Analysis
An auto-regressive language model is being trained on a large text corpus. At one training step, the model processes the input 'The cat sat on the' and must predict the next token. The actual next token in the training data is 'mat'. Which of the following predicted probability distributions for the next token would result in the lowest cross-entropy loss?