Example of a User Prompt in RLHF

Training a Reward Model with Preference Data

Techniques for Generating Diverse Outputs in RLHF

A team is developing a system to align a language model with human preferences. Their data collection process involves providing a prompt to an existing, fine-tuned model, which then generates a single response. A human labeler then assigns a quality score from 1 to 10 to this single response. This process is repeated for thousands of different prompts. What is the most significant flaw in this methodology for the purpose of creating a robust preference-based reward model?

Arrange the following steps in the correct chronological order to describe the data collection process for training a reward model.

Designing a Data Collection Pipeline for a Creative Writing Assistant