Learn Before
Applying a Preference Model for AI Fine-Tuning
A development team is fine-tuning a large language model to be a better conversational assistant. They have already collected a dataset of human preferences, where evaluators chose the better of two model-generated responses for thousands of different prompts. Using this data, they have successfully trained a 'reward model' that accurately predicts a scalar score representing how much a human would likely prefer a given response. The team is now ready for the final stage of the process: using this reward model to update the conversational assistant itself. What is the primary goal of this final stage, and how is the scalar score from the reward model utilized to achieve this goal?
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is being fine-tuned using a reinforcement learning approach that incorporates human feedback. Arrange the following key stages of this process into the correct chronological order.
A team is fine-tuning a language model using a reinforcement learning process guided by human feedback. They observe that while the model's policy is successfully optimized to achieve high scores from its internal reward signal, the generated text is often repetitive, nonsensical, and misaligned with the original human preferences. Which of the following is the most likely cause of this discrepancy?
Applying a Preference Model for AI Fine-Tuning