Pair-wise Ranking Loss Formula for RLHF Reward Model

Empirical Reward Model Loss Formula using Bradley-Terry Model

A reward model is trained to learn human preferences by minimizing the following loss function, which is an expectation over a preference dataset $\mathcal{D}$:

$\mathcal{L}(\phi) = -\mathbb{E}_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\sim\mathcal{D}} [\log \text{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})]$

In this dataset, $\mathbf{y}_a$ represents a response preferred over response $\mathbf{y}_b$ for a given input $\mathbf{x}$. What is the primary effect of successfully minimizing this loss function on the model's behavior?

Reward Model Training Diagnosis

Composition of Reward Model Parameters (ϕ)

Approximating Expected Loss with Empirical Loss

Empirical Reward Model Loss Formula

Impact of Prediction Confidence on Reward Model Loss

Listwise Loss Formula from Accumulated Pairwise Comparisons

Empirical Reward Model Loss Formula

Empirical Formulation of Pair-wise Ranking Loss

A system learns a function, r(input, response), that assigns a numerical score indicating the quality of a response for a given input. The probability that response Y_a is preferred over response Y_b is then calculated using the formula: Probability = Sigmoid(r(input, Y_a) - r(input, Y_b)), where Sigmoid(z) = 1 / (1 + e^-z). Given the following scenarios for a single input, which one presents a logical inconsistency between the assigned scores and the resulting preference probability?

Preference Probability Calculation

Invariance of Preference Probability