PPO Objective for LLM Training

Derivation of the KL Divergence Objective for Policy Optimization

During the policy optimization stage of training a large language model, an engineer observes that the model's outputs are coherent and safe, but they show very little improvement over the initial supervised fine-tuned version and consistently receive mediocre scores from the reward model. Which of the following is the most likely cause of this issue, based on the policy optimization objective function that balances maximizing rewards with a penalty for policy divergence?

Analyzing the Trade-off in Policy Optimization

Analyzing a Modified Policy Optimization Objective