Learn Before
Reference Policy Definition in RLHF
The reference policy in Reinforcement Learning from Human Feedback (RLHF), denoted as , is defined as a probability distribution, , which is determined by the parameters of the reference model, . The mathematical expression for this definition is .
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Architecture and Function of the RLHF Value Model
Target Model (Policy Model) in RLHF
Reference Policy Definition in RLHF
Architecture and Function of the RLHF Reward Model
A development team is building a system to align a large language model using reinforcement learning from human feedback. Their setup includes a target model for text generation, a reference model, a reward model to score outputs based on human preferences, and a value model to predict future rewards. For computational efficiency, they decide to build the reward model using a Convolutional Neural Network (CNN) and the value model using a Recurrent Neural Network (RNN), while keeping the target a
LLM as the Agent in RLHF
An alignment process for a large language model uses a system composed of four distinct models, all sharing a common underlying architecture. Match each model component with its primary role in this system.
Architectural Consistency in Feedback-Based LLM Alignment
In a typical system for aligning a language model with human feedback, it is common practice to use a Transformer-based architecture for the text-generating models, while employing simpler, non-Transformer architectures for the reward and value models to reduce computational overhead.
Learn After
In a reinforcement learning process that uses human feedback, a 'reference model' with a fixed set of parameters, , is used as a baseline. For a specific input prompt, this model calculates that the probability of generating the word 'consequently' as the next word is 0.04. Given that the reference policy, , is formally defined as the probability distribution generated by this reference model, what is the value of $\pi_{\theta_{\text{ref}}}(
True or False: In a Reinforcement Learning from Human Feedback (RLHF) system, the reference policy is a function that is trained to approximate the probability distribution generated by the reference model.
Reference Policy and Model Probability