PPO Objective Formula for LLM Training in RLHF

An autoregressive language model is generating the two-token response 'Good day' given a prompt. The table below shows the per-token log-probabilities from the current policy being trained ($ext{Pr}_{ heta}$) and a fixed reference policy ($ext{Pr}_{ heta_{ ext{ref}}}$). The policy divergence penalty is calculated as the sum of the differences between the log-probabilities of the current and reference policies for each token.

| Token | $\log \text{Pr}_{\theta}(y_t|\dots)$ | $\log \text{Pr}_{\theta_{\text{ref}}}(y_t|\dots)$ | | :--- | :---: | :---: | | 'Good' | -0.8 | -1.5 | | 'day' | -0.4 | -2.1 |

Based on this data, what can be concluded about the current policy's behavior for this specific generation?

Diagnosing Training Issues with Policy Divergence

Overall PPO Objective Function for Language Models

Interpreting the Policy Divergence Penalty