In a policy-based language model alignment process, the reward r(x, y) for a response y to a prompt x is defined by the equation: $r(\mathbf{x}, \mathbf{y}) = \beta \left( \log \frac{\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})} + \log Z(\mathbf{x}) \right)$ where π_θ is the target policy, π_θ_ref is the reference policy, β is a positive scaling factor, and Z(x) is a normalization factor. If, for a specific response y_1, the target pol

In a framework for aligning language models, a reward function is defined as: $r(\mathbf{x}, \mathbf{y}) = \beta \left( \log \frac{\pi_{\theta}(\mathbf{y}|\mathbf{x})}{\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})} + \log Z(\mathbf{x}) \right)$ where $\pi_{\theta}$ is the target policy, $\pi_{\theta_{\text{ref}}}$ is a reference policy, $\beta$ is a scaling factor, and $Z(\mathbf{x})$ is a normalization factor dependent on the prompt $\mathbf{x}$. Given two distinct responses, $\mathbf{y}

Derivation of DPO Preference Probability from Policy Ratios

Analysis of Reward Function under Policy Convergence