Role of Regularization in Mitigating Reward Model Underdetermination

A reward model is being trained using a loss function that includes a regularization term to prevent its output scores from growing excessively large. The regularization component for a single pair of responses, $(\mathbf{y}_a, \mathbf{y}_b)$, to an input, $\mathbf{x}$, is calculated as $(r(\mathbf{x}, \mathbf{y}_a) + r(\mathbf{x}, \mathbf{y}_b))^2$, where $r$ is the reward score. A higher value for this term results in a larger penalty. Given the following four pairs of reward scores, which pair would incur the largest penalty from this specific regularization term?

A reward model is being trained with a loss function that includes a regularization component. This component adds a penalty proportional to $(r(\mathbf{x}, \mathbf{y}_a) + r(\mathbf{x}, \mathbf{y}_b))^2$ for a given input $\mathbf{x}$ and a pair of responses $(\mathbf{y}_a, \mathbf{y}_b)$. The goal of this penalty is to prevent reward scores from becoming excessively large. Consider two scenarios for the reward scores assigned to a pair of responses:

• Scenario 1: $r(\mathbf{x}, \mathbf{y}_a) = 10$ and $r(\mathbf{x}, \mathbf{y}_b) = -10$
• Scenario 2: $r(\mathbf{x}, \mathbf{y}_a) = 5$ and $r(\mathbf{x}, \mathbf{y}_b) = 5$

Based on the formula for the penalty, which of the following statements correctly analyzes the effect of the regularization in these two scenarios?

Diagnosing Reward Model Score Inflation