An autoregressive language model is generating the two-token response 'Good day' given a prompt. The table below shows the per-token log-probabilities from the current policy being trained ($ext{Pr}_{ heta}$) and a fixed reference policy ($ext{Pr}_{ heta_{ ext{ref}}}$). The policy divergence penalty is calculated as the sum of the differences between the log-probabilities of the current and reference policies for each token.

| Token | $\log \text{Pr}_{\theta}(y_t|\dots)$ | $\log \text{Pr}_{\t

Diagnosing Training Issues with Policy Divergence

Overall PPO Objective Function for Language Models

Interpreting the Policy Divergence Penalty