Google

A reference policy, denoted $$\pi_{\theta_{\text{ref}}}$$, is a fixed policy used as a baseline in certain reinforcement learning algorithms. It is typically an earlier version of the policy being trained or a pre-trained model. The training process aims to improve the current policy ($$\pi_\theta$$) without deviating too far from the reference policy, which helps stabilize learning.

Reference Policy ($$\pi_{\theta_{\text{ref}}}$$)

Imagine two separate training processes for a language model designed to be a helpful assistant. 

*   **Process A:** The model is updated based solely on maximizing rewards from user feedback for the assistant task.
*   **Process B:** The model is updated to maximize rewards from user feedback, but with an added constraint: it is penalized if its response probabilities deviate significantly from those of its original, general-purpose, pre-trained version.

Analyze the potential trade-offs between these two training processes. In your analysis, discuss the likely effects on the final model's performance, stability during training, and retention of its initial capabilities.

Analysis of Constrained vs. Unconstrained Model Training

During the fine-tuning of a large language model, an engineer observes that the model's outputs are rapidly degrading, becoming nonsensical and repetitive after only a few training steps. Briefly explain how introducing a fixed, pre-trained version of the model as a baseline to compare against during the training process could mitigate this issue.

Stabilizing Model Fine-Tuning

In Reinforcement Learning from Human Feedback (RLHF), the reference policy, denoted as $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})$$, is a fixed policy used as a baseline during the optimization of the active policy $$\pi_{\theta}$$. It is typically a copy of the supervised fine-tuned (SFT) model before the RLHF stage begins. The reference policy's role is to prevent the active policy from deviating too far from the original language style and safety constraints, which is enforced by a penalty term (e.g., KL-divergence) that measures the difference between the two policies.

Reference Policy in RLHF

A team is fine-tuning a large language model (the 'active model') to improve its performance on a specific task. They use the original, pre-trained version of the model as a fixed baseline. During training, a penalty is applied to the active model whenever its output probabilities for generating the next piece of text diverge significantly from the baseline model's probabilities. What is the most likely reason for incorporating this penalty mechanism?

Learn Before

Related