Google

The pair-wise ranking loss function for a reward model with parameters $$\omega$$ can be formulated by summing over samples from the preference dataset $$\mathcal{D}_r$$. The expected loss incorporates the probability $$\Pr(\mathbf{x})$$ of drawing an input, and the conditional probability $$\Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x})$$ of drawing the preferred output pair. Assuming a uniform distribution over $$K$$ model inputs involved in sampling, the formula simplifies to:

$$\begin{aligned} \mathrm{Loss}_{\omega}(\mathcal{D}_r) &= -\sum \Pr(\mathbf{x}) \cdot \Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x}) \cdot \log(\mathrm{Sigmoid}(R_{\omega}(\mathbf{x}, \mathbf{y}_{k_1}) - R_{\omega}(\mathbf{x}, \mathbf{y}_{k_2}))) \\ &= -\frac{1}{K} \sum \Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x}) \cdot \log(\mathrm{Sigmoid}(R_{\omega}(\mathbf{x}, \mathbf{y}_{k_1}) - R_{\omega}(\mathbf{x}, \mathbf{y}_{k_2}))) \end{aligned}$$

Empirical Formulation of Pair-wise Ranking Loss

A team is training a reward model using preference data. They are considering two different loss formulations for a single data sample consisting of a prompt `x`, a preferred response `y_pref`, and a rejected response `y_rej`.

**Loss Formulation A:**
`Loss = -log(Sigmoid(R(x, y_pref) - R(x, y_rej)))`

**Loss Formulation B:**
`Loss = -Pr(y_pref ≻ y_rej | x) * log(Sigmoid(R(x, y_pref) - R(x, y_rej)))`

Explain the practical implication of including the `Pr(y_pref ≻ y_rej | x)` term in Loss Formulation B. How does this term change the way the model learns from the preference data compared to using Loss Formulation A?

Analysis of a Weighted Ranking Loss

Based on the empirical formulation of the pair-wise ranking loss, which incorporates preference probabilities, explain how the 70/30 split in labeler preference for this specific data point influences the loss calculation and the subsequent update to the model's parameters. Contrast this with a scenario where all 10 labelers agreed that `y_A` was the preferred response.

Handling Labeler Disagreement in Reward Modeling

A reward model is being trained using a pair-wise ranking loss function. For a given prompt `x`, the preference dataset contains a pair of responses: a preferred response `y_pref` and a rejected response `y_rej`. Initially, the model assigns the following scores: `R(x, y_pref) = 2.0` and `R(x, y_rej) = 3.0`. Based on the objective of the loss function, what is the most likely change to these scores after a single optimization step on this data point?

Learn Before

Related