logo
How it worksResearch CommunitiesBenefitsAbout Us
Schedule Demo
Learn Before

  • Listwise Loss from Accumulated Pairwise Comparisons

  • Modeling Pairwise Preference Probability with a Reward Function

Listwise Loss Formula from Accumulated Pairwise Comparisons

The listwise loss, derived from aggregating pairwise comparisons, is formally defined as the negative expected log-likelihood over all distinct pairs in a ranked list. The formula is:

Llist=−E(x,Y)∼Dr[1N(N−1)∑ya∈Y,yb∈Yya≠yblog⁡Pr⁡(ya≻yb∣x)]\mathcal{L}_{\text{list}} = -\mathbb{E}_{(\mathbf{x},Y)\sim\mathcal{D}_r}\left[\frac{1}{N(N-1)}\sum_{\substack{\mathbf{y}_a\in Y, \mathbf{y}_b\in Y \\ \mathbf{y}_a\neq \mathbf{y}_b}} \log\Pr(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})\right]

Here:

  • Llist\mathcal{L}_{\text{list}} is the listwise loss.
  • The expectation E\mathbb{E} is taken over samples (x,Y)(\mathbf{x}, Y) from the preference dataset Dr\mathcal{D}_r, where YY is the ranked list of NN outputs for a prompt x\mathbf{x}.
  • The summation aggregates the log probability of the ground-truth preference for every ordered pair of distinct outputs (ya,yb)(\mathbf{y}_a, \mathbf{y}_b) within the list YY.
  • The term 1N(N−1)\frac{1}{N(N-1)} serves as a normalization factor, averaging the loss over the total number of possible ordered pairs.
Image 0

0

1

14 days ago

Contributors are:

Gemini AI
Gemini AI
🏆 10

Who are from:

Google
Google
🏆 10

References

Tags

Ch.4 Alignment - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Related

  • Listwise Loss Formula from Accumulated Pairwise Comparisons

  • A human annotator is given four model-generated responses (A, B, C, D) to a prompt and ranks them in order of preference from best to worst as: C > A > D > B. To train a preference model, a loss function is calculated by summing the individual losses for every pairwise comparison implied by this ranking. Which of the following sets represents all the pairwise preferences that would be used in this loss calculation?

  • Decomposing a Ranked List into Pairwise Preferences

  • Evaluating Preference Model Performance with Listwise Loss

  • Listwise Loss Formula from Accumulated Pairwise Comparisons

  • Empirical Reward Model Loss Formula

  • Empirical Formulation of Pair-wise Ranking Loss

  • A system learns a function, r(input, response), that assigns a numerical score indicating the quality of a response for a given input. The probability that response Y_a is preferred over response Y_b is then calculated using the formula: Probability = Sigmoid(r(input, Y_a) - r(input, Y_b)), where Sigmoid(z) = 1 / (1 + e^-z). Given the following scenarios for a single input, which one presents a logical inconsistency between the assigned scores and the resulting preference probability?

  • Preference Probability Calculation

  • Invariance of Preference Probability

Learn After

  • Consider the following formula for a loss function used to train a model on ranked lists of outputs, where N is the number of items in a given list Y:

    L=−E[1N(N−1)∑ya∈Y,yb∈Yya≠yblog⁡Pr⁡(ya≻yb∣x)]\mathcal{L} = -\mathbb{E}\left[\frac{1}{N(N-1)}\sum_{\substack{\mathbf{y}_a\in Y, \mathbf{y}_b\in Y \\ \mathbf{y}_a\neq \mathbf{y}_b}} \log\Pr(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})\right]

    What is the primary analytical consequence of including the normalization term 1N(N−1)\frac{1}{N(N-1)} in this calculation?

  • Applying the Listwise Loss Summation

  • Consider the listwise loss formula used for training on ranked preferences:

    L=−E[1N(N−1)∑ya,yb∈Yya≠yblog⁡Pr⁡(ya≻yb∣x)]\mathcal{L} = -\mathbb{E}\left[\frac{1}{N(N-1)}\sum_{\substack{\mathbf{y}_a, \mathbf{y}_b \in Y \\ \mathbf{y}_a\neq \mathbf{y}_b}} \log\Pr(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})\right]

    True or False: If a model is completely uncertain about the preferences within a ranked list (i.e., it assigns Pr⁡(ya≻yb∣x)=0.5\Pr(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x}) = 0.5 for all distinct pairs), the contribution of that specific list to the overall loss will be zero.