An AI development team is refining a pre-trained language model using a dataset of human preferences, where each example consists of a prompt, a preferred response, and a rejected response. As training progresses, they notice that while the model is learning to generate responses that align with the preferences, its general language quality is deteriorating; it produces more repetitive and nonsensical text. What is the most probable cause of this issue related to the optimization objective's design?

Choosing a Baseline for Preference Alignment

Selecting a Baseline for Policy Optimization

Conceptual Objective Function Assumed in DPO