Google

Strategy 1: The team uses a highly aligned group of labelers who almost always agree. For 95% of the prompts, at least 9 out of 10 labelers choose the same response as the &#x27;winner&#x27;.
Strategy 2: The team uses a more diverse group of labelers. For

Research indicates that having significant variability within the pairwise preference data is a key factor for successfully training Large Language Models, regardless of whether the feedback originates from humans or AI systems.

Importance of Variability in Pairwise Preference Data

Given the following scenario, identify the most probable cause of the model's training plateau and recommend a single change to the data collection process to address it.

Diagnosing a Model Training Plateau

An AI training team has two datasets of pairwise preference feedback for fine-tuning a language model. Both datasets have the same number of prompts.

*   **Dataset X:** For the vast majority of prompts, the labelers showed very high agreement, with over 95% choosing the same response as the 'winner'.
*   **Dataset Y:** For many prompts, the labelers showed significant disagreement, with preferences often split closer to 60/40 or 70/30.

Which dataset is likely to be more valuable for training a sophisticated and nuanced language model? Justify your choice by explaining the role of preference distribution in the training process.

Evaluating Preference Datasets

A team is training a language model using preference data from a group of 10 labelers. For each prompt, the labelers are shown two potential responses and asked to choose the better one. The team considers two data collection strategies:

*   **Strategy 1:** The team uses a highly aligned group of labelers who almost always agree. For 95% of the prompts, at least 9 out of 10 labelers choose the same response as the 'winner'.
*   **Strategy 2:** The team uses a more diverse group of labelers. For

Learn Before

Related