Combining Multiple Reward Models to Mitigate Overoptimization
A practical strategy to address the overoptimization problem is to combine multiple reward models. This approach is considered more feasible than attempting to build a single, perfect oracle model. By aggregating feedback from several distinct models, the system can reduce the misalignment between the training objective and the true objective that often arises when relying on a single, imperfect reward model, leading to a more robust and accurate overall reward signal.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Learn After
Combining Reward Models as an Ensemble Learning Problem
Bayesian Model Averaging for Combining Reward Models
Fusion Networks for Combining Reward Models
Multi-Objective Optimization for Policy Training with Multiple Reward Models
Ensemble Learning Techniques for Reward Model Creation
Aspect-Based Reward Model Construction in RLHF
Using Off-the-Shelf LLMs as Reward Models
A team is training a language model to generate helpful cooking recipes. They use a single reward model that scores recipes based on the number of ingredients from a predefined 'healthy' list. They observe that the model starts generating nonsensical recipes that are just long lists of these healthy ingredients, achieving very high reward scores but being completely useless for cooking. Which of the following approaches is the most robust solution to prevent the model from exploiting the reward system in this way?
Reward System Design Strategy
Evaluating a Chatbot Training Strategy