Learn Before
Complexity of Reward Model Training in RLHF
While learning reward models is a standard component of reinforcement learning, it introduces significant complexity to the entire training process compared to standard supervised training methods. Developing a reliable reward model is inherently difficult, and failing to do so—resulting in a poorly trained reward model—can severely impact and degrade the overall outcome of policy learning.
0
1
Tags
Foundations of Large Language Models
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Policy Learning in RLHF
Dual Role of the RLHF Reward Model: Ranking-based Training for Scoring Application
Relation between Verifiers and RLHF Reward Models
General Loss Minimization Objective for Reward Model Training
Architecture and Function of the RLHF Reward Model
Reward Model Training as a Ranking Problem in RLHF
Underdetermined Model
Limitations of Outcome-Based Rewards for Entire Sequences
Training a Reward Model with Preference Data
Converting Listwise Rankings to Pairwise Preferences for Reward Model Training
Diagnosing Undesired Model Behavior
An AI team is training a reward model using a dataset where, for each prompt, human annotators have ranked several generated responses from best to worst. What is the fundamental task the reward model is being trained to perform based on this specific type of data?
An AI development team is training a model to act as a helpful assistant. They create a dataset where, for each user prompt, human evaluators are shown two different generated responses and asked to choose which one is better. The model is then trained on this dataset of pairwise preferences. After training, the team observes that the model consistently assigns higher scores to longer, more detailed responses, even when they are less helpful or contain irrelevant information. Which of the following is the most likely explanation for this emergent behavior?
Ranking LLM Outputs as an Alternative to Rating
Regularization in RLHF Reward Model Training
Complexity of Reward Model Training in RLHF