Justifying the Reference Policy
A language model's policy is being shaped using a reward function, . Instead of defining the policy as being directly proportional to only the exponentiated reward, it is instead defined as proportional to the product of a reference policy and the exponentiated reward: . Explain the primary benefit of including the term in this formulation.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is refining a text-generation model. The final probability of a generated text sequence is proportional to the product of its probability from an initial base model and an exponentiated reward score. The reward's influence is controlled by a scaling parameter, β, in the exponent, where a smaller β gives the reward more weight. The team observes that when they significantly decrease the value of β, the model's outputs become more repetitive and sometimes nonsensical, even though they achieve very high scores from the reward model. Which of the following best explains this behavior?
Diagnosing Language Model Alignment Issues
Justifying the Reference Policy