Establishing the Initial Policy in RLHF
The starting point for Reinforcement Learning from Human Feedback (RLHF) is an initial policy, which is an LLM that has already undergone pre-training and instruction fine-tuning. This model is considered the version that would be deployed to interact with users and respond to their requests, forming the baseline for further alignment.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Establishing the Initial Policy in RLHF
A team is developing a language model designed to align with human preferences. They are following a standard four-stage process. Arrange the following stages in the correct chronological order.
A development team is using a four-stage process to align a language model with human preferences. They collect a large dataset where human annotators consistently rank verbose and evasive responses as low quality. This dataset is then used to train a reward model. Finally, the language model is fine-tuned using reinforcement learning, with the reward model providing the optimization signal. However, the final, aligned language model still frequently produces verbose and evasive outputs. Which stage is the most likely source of this failure?
A team is aligning a language model with human preferences using a four-stage process. Match each stage of the process to its primary function and the key artifact it produces.
Learn After
A development team is preparing to use a human feedback-driven process to improve an AI's helpfulness and safety. They have two candidate models to use as their starting point:
Model A: A raw, pre-trained model that is very good at predicting the next word in a sentence but has not been specifically trained to follow user commands.
Model B: A model that has been pre-trained and then further fine-tuned on a dataset of instructions and high-quality answers, making it proficient at following user commands.
Which statement best evaluates the choice of a starting model for this alignment process?
Diagnosing an Inefficient Alignment Process
Characteristics of the Starting Model for Alignment