Role of Regularization in Mitigating Reward Model Underdetermination

A reward model is being trained using a loss function that includes a regularization term to prevent its output scores from growing excessively large. The regularization component for a single pair of responses, $(\mathbf{y}_a, \mathbf{y}_b)$, to an input, $\mathbf{x}$, is calculated as $(r(\mathbf{x}, \mathbf{y}_a) + r(\mathbf{x}, \mathbf{y}_b))^2$, where $r$ is the reward score. A higher value for this term results in a larger penalty. Given the following four pairs of reward scores, which pai

A reward model is being trained with a loss function that includes a regularization component. This component adds a penalty proportional to $(r(\mathbf{x}, \mathbf{y}_a) + r(\mathbf{x}, \mathbf{y}_b))^2$ for a given input $\mathbf{x}$ and a pair of responses $(\mathbf{y}_a, \mathbf{y}_b)$. The goal of this penalty is to prevent reward scores from becoming excessively large. Consider two scenarios for the reward scores assigned to a pair of responses:

• Scenario 1: $r(\mathbf{x}, \mathbf{y}_a)

Diagnosing Reward Model Score Inflation