A team is refining a text-generation model. The final probability of a generated text sequence is proportional to the product of its probability from an initial base model and an exponentiated reward score. The reward's influence is controlled by a scaling parameter, β, in the exponent, where a smaller β gives the reward more weight. The team observes that when they significantly decrease the value of β, the model's outputs become more repetitive and sometimes nonsensical, even though they achieve very high scores from the reward model. Which of the following best explains this behavior?

Diagnosing Language Model Alignment Issues

Justifying the Reference Policy