Analyzing the Impact of a Policy Divergence Penalty
When training a language model, a penalty term is often added to the learning objective to limit how much the model's behavior can change in a single update, measured against a reference version of the model. Analyze the potential trade-offs involved in this approach. Specifically, discuss the likely consequences for the training process and final model performance if the weight of this penalty is set (a) too high, and (b) too low.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
An engineer is fine-tuning a language model and observes that the training process is highly unstable. The model's performance fluctuates wildly, and the training loss sometimes spikes dramatically, suggesting the policy updates are too aggressive. Which of the following modifications to the optimization objective is most specifically designed to counteract this problem by directly constraining the magnitude of policy changes at each step?
Stabilizing an Erratic Training Process
Analyzing the Impact of a Policy Divergence Penalty