A language model is being fine-tuned on a dataset D containing two input-output pairs: (x1, y1) and (x2, y2). The training objective is to find the model parameters that maximize the sum of the log-probabilities of the concatenated input-output sequences across the entire dataset.

Two candidate models, Model A and Model B, produce the following log-probabilities for the concatenated sequences:

• Model A:
  • log Pr(seq_x1,y1) = -1.2
  • log Pr(seq_x2,y2) = -0.8
• Model B:
  • log Pr(seq_x1,y1) = -0.9
  • log Pr(seq_x2,y2) = -1.3

Based on the stated training objective, which model is preferred and why?

Interpreting the SFT Objective Function

When fine-tuning a language model with the objective to maximize the sum of log-probabilities across all concatenated input-output sequences in a dataset, which of the following statements accurately describes the training dynamics?