Combining Reward Models as an Ensemble Learning Problem

Bayesian Model Averaging for Combining Reward Models

Fusion Networks for Combining Reward Models

Multi-Objective Optimization for Policy Training with Multiple Reward Models

Ensemble Learning Techniques for Reward Model Creation

Aspect-Based Reward Model Construction in RLHF

Using Off-the-Shelf LLMs as Reward Models

A team is training a language model to generate helpful cooking recipes. They use a single reward model that scores recipes based on the number of ingredients from a predefined 'healthy' list. They observe that the model starts generating nonsensical recipes that are just long lists of these healthy ingredients, achieving very high reward scores but being completely useless for cooking. Which of the following approaches is the most robust solution to prevent the model from exploiting the reward system in this way?

Reward System Design Strategy

Evaluating a Chatbot Training Strategy