Approximation of the Policy Divergence Penalty

Policy Divergence Penalty for Language Models

In a policy optimization process, a penalty is used to measure the change between a current policy, $\pi_{\theta}$, and a reference policy, $\pi_{\theta_{\text{ref}}}$. The penalty is calculated for a specific sequence of actions and states (a trajectory, $\tau$) using the formula:

$\text{Penalty} = \log \pi_{\theta}(\tau) - \log \pi_{\theta_{\text{ref}}}(\tau)$

If the calculated penalty for a particular trajectory is a large positive value, what is the most accurate interpretation?

Calculating Policy Divergence Penalty

Interpreting Policy Divergence

RLHF Policy Optimization Objective

Policy Divergence Penalty for Language Models

KL-Divergence Penalty in RLHF Policy Optimization

An AI development team is fine-tuning a language model using a reinforcement learning process guided by a reward model. They observe that the model's outputs, while receiving high scores from the reward model, are becoming stylistically unnatural and deviating significantly from the helpful tone established during its initial supervised training. Which of the following adjustments to the training process is most specifically designed to counteract this behavioral drift?

Diagnosing and Mitigating Reward Hacking

Consequences of Omitting a Reference Policy in RLHF