Case Study

Improving a Preference Labeling Prompt

A research team is creating a preference dataset to improve an AI's ability to generate helpful and harmless responses. They provide human labelers with two AI-generated responses to a user's question and use the following prompt to collect the data:

Prompt: 'Here are two responses to a user's question. Which one is better? Choose A or B.'

After an initial round of data collection, the team observes that the labelers' choices are highly inconsistent, making the resulting dataset unreliable. Based on your understanding of how to generate high-quality preference data, evaluate the team's prompt. Identify its most significant weakness and propose a specific modification that incorporates a reasoning-based prompting technique to improve the consistency and quality of the labels. Explain why your proposed modification would be effective.

0

1

Updated 2025-09-28

Contributors are:

Who are from:

Tags

Ch.5 Inference - Foundations of Large Language Models

Foundations of Large Language Models

Foundations of Large Language Models Course

Computing Sciences

Evaluation in Bloom's Taxonomy

Cognitive Psychology

Psychology

Social Science

Empirical Science

Science