Learn Before
Consequences of Omitting a Reference Policy in RLHF
Imagine a team is training a large language model using a reinforcement learning process. They have a reward model that accurately scores outputs for helpfulness. However, they decide to optimize their active policy to maximize this reward directly, without comparing it to a fixed, initial version of the model. Analyze the potential negative consequences of this approach. Describe at least two distinct undesirable behaviors the final model might exhibit and explain why these behaviors could arise in the absence of this comparison.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
RLHF Policy Optimization Objective
Policy Divergence Penalty for Language Models
KL-Divergence Penalty in RLHF Policy Optimization
An AI development team is fine-tuning a language model using a reinforcement learning process guided by a reward model. They observe that the model's outputs, while receiving high scores from the reward model, are becoming stylistically unnatural and deviating significantly from the helpful tone established during its initial supervised training. Which of the following adjustments to the training process is most specifically designed to counteract this behavioral drift?
Diagnosing and Mitigating Reward Hacking
Consequences of Omitting a Reference Policy in RLHF