PPO Objective Formula for LLM Training in RLHF
The policy in RLHF is updated by minimizing the Proximal Policy Optimization (PPO) loss. This objective function combines a clipped surrogate objective, which uses the advantage function , with a penalty term to prevent large deviations from the reference policy (). The formula is expressed as: This loss is minimized over all prompts in the dataset and for each token in the generated sequence . The term scaled by acts as a KL-divergence penalty to ensure training stability.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A language model is being trained using an objective function that balances a reward-based component with a penalty for deviating from an initial reference policy. The penalty's influence is controlled by a coefficient, β. During training, developers observe that the model's outputs, while achieving high reward scores, are becoming increasingly repetitive and nonsensical. Which of the following adjustments to β is the most appropriate first step to mitigate this issue, and why?
Impact of Penalty Coefficient on LLM Fine-Tuning
Consequences of Modifying the PPO Objective Function
PPO Objective Formula for LLM Training in RLHF
Learn After
Diagnosing LLM Training Instability
A team is fine-tuning a large language model using a reinforcement learning objective that includes a clipped probability ratio multiplied by an advantage estimate, and a penalty term based on the divergence from a reference model. During training, they observe that while the model's average reward is increasing, its outputs are becoming nonsensical and repetitive, losing the general language capabilities of the original model. Which of the following is the most likely cause of this issue?
A language model is being trained using a reinforcement learning objective. For each generated token, part of this objective is calculated as:
Clip(probability_ratio) * Advantage. Theprobability_ratiois the likelihood of generating the token under the new policy divided by the likelihood under the old policy, andAdvantageis an estimate of how much better that token was than the expected average. In a particular training step for a tokeny, theAdvantageis strongly positive, and theprobability_ratiois already high (e.g., 1.5, where the clipping threshold is 1.2). How does theClipfunction influence the update to the model's policy for generating tokeny?