A model is being trained to evaluate text completions for a given prompt. The training data consists of pairs of completions for each prompt, where one is marked as 'preferred' ($y_a$) and the other as 'dispreferred' ($y_b$) by human reviewers. The model learns by minimizing the following loss function, averaged over all pairs in the dataset:

$\mathcal{L} = - \log \sigma(score(y_a) - score(y_b))$

where $\sigma$ is the sigmoid function and $score(y)$ is the value the model assigns to a completion $y$.

What is the primary effect of minimizing this loss function on the scores the model assigns?

Calculating Pair-wise Ranking Loss

Analyzing Reward Model Loss Behavior