RLHF Policy Optimization Objective

Policy Divergence Penalty for Language Models

KL-Divergence Penalty in RLHF Policy Optimization

An AI development team is fine-tuning a language model using a reinforcement learning process guided by a reward model. They observe that the model's outputs, while receiving high scores from the reward model, are becoming stylistically unnatural and deviating significantly from the helpful tone established during its initial supervised training. Which of the following adjustments to the training process is most specifically designed to counteract this behavioral drift?

Diagnosing and Mitigating Reward Hacking

Consequences of Omitting a Reference Policy in RLHF