Establishing the Initial Policy in RLHF
The starting point for Reinforcement Learning from Human Feedback (RLHF) is an initial policy, which is an LLM that has already undergone pre-training and instruction fine-tuning. This model is considered the version that would be deployed to interact with users and respond to their requests, forming the baseline for further alignment.
0
1
Tags
Ch.2 Generative Models - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Establishing the Initial Policy in RLHF
A team is developing a language model designed to align with human preferences. They are following a standard four-stage process. Arrange the following stages in the correct chronological order.
A development team is using a four-stage process to align a language model with human preferences. They collect a large dataset where human annotators consistently rank verbose and evasive responses as low quality. This dataset is then used to train a reward model. Finally, the language model is fine-tuned using reinforcement learning, with the reward model providing the optimization signal. However, the final, aligned language model still frequently produces verbose and evasive outputs. Which s
A team is aligning a language model with human preferences using a four-stage process. Match each stage of the process to its primary function and the key artifact it produces.
Learn After
A development team is preparing to use a human feedback-driven process to improve an AI's helpfulness and safety. They have two candidate models to use as their starting point:
Model A: A raw, pre-trained model that is very good at predicting the next word in a sentence but has not been specifically trained to follow user commands.
Model B: A model that has been pre-trained and then further fine-tuned on a dataset of instructions and high-quality answers, making it proficient at following use
Diagnosing an Inefficient Alignment Process
Characteristics of the Starting Model for Alignment