SFT Objective as Maximizing Joint Log-Probability of Concatenated Sequences
When Supervised Fine-Tuning (SFT) is framed as a standard language model training task, the objective is to find the parameters that maximize the sum of the log-probabilities of the concatenated input-output sequences across the entire dataset . This is formally expressed as: . By taking as the objective function, SFT can be implemented using standard LLMs, treating the combined input and output as a single sequence for the model to process.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
SFT Objective as Maximizing Joint Log-Probability of Concatenated Sequences
In a common fine-tuning strategy, a prompt and its desired completion are concatenated into a single sequence (e.g.,
[prompt_tokens, completion_tokens]). The language model is then trained on this full sequence, but the training loss is calculated only for the model's predictions on the completion tokens. What is the most accurate analysis of the primary purpose of this specific loss calculation method?During supervised fine-tuning, if a model is trained on concatenated
[input, output]sequences and the training loss is calculated across the entire sequence (both input and output tokens), the model is still being optimized primarily to improve its conditional generation capabilities for the given input.Diagnosing a Faulty Fine-Tuning Process
Loss Masking via Forward and Backward Passes in SFT
Learn After
A language model is being fine-tuned on a dataset
Dcontaining two input-output pairs:(x1, y1)and(x2, y2). The training objective is to find the model parameters that maximize the sum of the log-probabilities of the concatenated input-output sequences across the entire dataset.Two candidate models, Model A and Model B, produce the following log-probabilities for the concatenated sequences:
- Model A:
log Pr(seq_x1,y1) = -1.2log Pr(seq_x2,y2) = -0.8
- Model B:
log Pr(seq_x1,y1) = -0.9log Pr(seq_x2,y2) = -1.3
Based on the stated training objective, which model is preferred and why?
- Model A:
Interpreting the SFT Objective Function
When fine-tuning a language model with the objective to maximize the sum of log-probabilities across all concatenated input-output sequences in a dataset, which of the following statements accurately describes the training dynamics?