Approximation of the Policy Divergence Penalty

Policy Divergence Penalty for Language Models

In a policy optimization process, a penalty is used to measure the change between a current policy, $\pi_{\theta}$, and a reference policy, $\pi_{\theta_{\text{ref}}}$. The penalty is calculated for a specific sequence of actions and states (a trajectory, $\tau$) using the formula:

$\text{Penalty} = \log \pi_{\theta}(\tau) - \log \pi_{\theta_{\text{ref}}}(\tau)$

If the calculated penalty for a particular trajectory is a large positive value, what is the most accurate interpretation?

Calculating Policy Divergence Penalty

Interpreting Policy Divergence