Reward Function in Terms of Policy Models and Normalization Factor
By rearranging the equation for the optimal target policy, the underlying reward function can be expressed solely using the target model , the reference model , and the normalization factor . This is a profound shift because, although the initial goal was to learn a policy using a given reward model, it leads to a representation of the reward model derived entirely from the policy. The resulting formula is:
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Reward Function in Terms of Policy Models and Normalization Factor
In a particular policy optimization framework, the target policy, denoted as , is determined by the following relationship involving a reference policy , a reward function , a positive temperature parameter , and a normalization term : $$ \pi_{\theta}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y}))}{Z(\math
Consider the following equation that defines a target policy based on a reference policy , a reward function , a positive scaling parameter , and a normalization term : True or False: If the reward function is equal to
Impact of the Scaling Parameter on Policy Behavior
Learn After
In a policy-based language model alignment process, the reward
r(x, y)for a responseyto a promptxis defined by the equation: whereπ_θis the target policy,π_θ_refis the reference policy,βis a positive scaling factor, andZ(x)is a normalization factor. If, for a specific responsey_1, the target polIn a framework for aligning language models, a reward function is defined as: where is the target policy, is a reference policy, is a scaling factor, and is a normalization factor dependent on the prompt . Given two distinct responses, $\mathbf{y}
Derivation of DPO Preference Probability from Policy Ratios
Analysis of Reward Function under Policy Convergence