Pair-wise Ranking Loss Formula for RLHF Reward Model

Empirical Reward Model Loss Formula using Bradley-Terry Model

A reward model is trained to learn human preferences by minimizing the following loss function, which is an expectation over a preference dataset $\mathcal{D}$:

$\mathcal{L}(\phi) = -\mathbb{E}_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\sim\mathcal{D}} [\log \text{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})]$

In this dataset, $\mathbf{y}_a$ represents a response preferred over response $\mathbf{y}_b$ for a given input $\mathbf{x}$. What is the primary effect of successfully minimizing this loss function on the model's behavior?

Reward Model Training Diagnosis

Composition of Reward Model Parameters (ϕ)

Approximating Expected Loss with Empirical Loss

Empirical Reward Model Loss Formula

Impact of Prediction Confidence on Reward Model Loss