Loss Masking via Forward and Backward Passes in SFT
In Supervised Fine-Tuning (SFT), training can be implemented using standard Large Language Models by concatenating the input and target output into a single sequence . During the forward pass, the model processes the entire sequence as usual. Then, during the backward pass, the loss corresponding to the input tokens is forced to zero (masked). This focuses the loss computation and subsequent parameter updates solely on the conditional probability of the output tokens, , while effectively setting the term to .
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
SFT Objective as Maximizing Joint Log-Probability of Concatenated Sequences
In a common fine-tuning strategy, a prompt and its desired completion are concatenated into a single sequence (e.g.,
[prompt_tokens, completion_tokens]). The language model is then trained on this full sequence, but the training loss is calculated only for the model's predictions on the completion tokens. What is the most accurate analysis of the primary purpose of this specific loss calculation method?During supervised fine-tuning, if a model is trained on concatenated
[input, output]sequences and the training loss is calculated across the entire sequence (both input and output tokens), the model is still being optimized primarily to improve its conditional generation capabilities for the given input.Diagnosing a Faulty Fine-Tuning Process
Loss Masking via Forward and Backward Passes in SFT
A language model is being trained on instruction-following data. For one specific training instance, the model processes the full tokenized sequence:
['User:', 'What', 'is', '2+2?', 'Assistant:', '4']. The goal is to train the model to provide the correct response ('4') when given the user's prompt. During the backpropagation step for this single instance, on which token(s) is the predictive loss calculated to update the model's weights?Diagnosing a Faulty Language Model Training Process
A machine learning engineer is training a language model for a question-answering task. The training data consists of concatenated
[question, answer]sequences. Due to a configuration error, the training loss is calculated across all tokens in the sequence (both question and answer), instead of only on the answer tokens. What is the most likely and significant negative consequence of this misconfiguration on the model's behavior?Loss Masking via Forward and Backward Passes in SFT
Learn After
A machine learning engineer is fine-tuning a pre-trained language model to function as a helpful assistant. The training data consists of pairs of instructions and desired responses. For each pair, the instruction and response are combined into a single sequence, and the model is trained to predict the next token at each position. However, due to a configuration error, the training loss is calculated across the entire combined sequence (both the instruction and the response tokens), instead of only on the response tokens. What is the most likely undesirable outcome of this training setup?
Applying Loss Masking in SFT
Analyzing a Fine-Tuning Training Objective