Evaluating Preference Datasets
An AI training team has two datasets of pairwise preference feedback for fine-tuning a language model. Both datasets have the same number of prompts.
- Dataset X: For the vast majority of prompts, the labelers showed very high agreement, with over 95% choosing the same response as the 'winner'.
- Dataset Y: For many prompts, the labelers showed significant disagreement, with preferences often split closer to 60/40 or 70/30.
Which dataset is likely to be more valuable for training a sophisticated and nuanced language model? Justify your choice by explaining the role of preference distribution in the training process.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is training a language model using preference data from a group of 10 labelers. For each prompt, the labelers are shown two potential responses and asked to choose the better one. The team considers two data collection strategies:
- Strategy 1: The team uses a highly aligned group of labelers who almost always agree. For 95% of the prompts, at least 9 out of 10 labelers choose the same response as the 'winner'.
- Strategy 2: The team uses a more diverse group of labelers. For many prompts, there is significant disagreement, with preferences often split 6-to-4 or 7-to-3.
Based on principles of effective model training, which strategy is likely to produce a more useful dataset, and why?
Diagnosing a Model Training Plateau
Evaluating Preference Datasets