Google

An effective way to improve preference labeling is to incorporate a Chain-of-Thought (CoT) rationale within the prompt. For instance, the prompt could include an example that not only states a preference for one response over another but also provides a step-by-step explanation for this choice, guiding the labeler to apply similar critical reasoning.

Example of Using CoT in a Preference Labeling Prompt

A team is collecting data to train a helpful AI assistant, but their human labelers are providing inconsistent quality ratings. The team suspects the instructional examples are the problem. Analyze the example provided to labelers in the case study below. Identify the primary weakness in the 'Reasoning' section and explain how you would rewrite it to provide a clearer, more effective, and repeatable analytical process for the labelers.

Diagnosing Inconsistent Preference Labeling

You are creating a set of instructions for data labelers. Below is a user query, two AI-generated responses, and a simple preference explanation. Your task is to rewrite the explanation to include a clear, step-by-step 'chain-of-thought' rationale that a labeler could use as a model for their own work.

**User Query:** "Explain the water cycle for a 5th grader."

**Response A:** "The water cycle is the process of water moving around the Earth. It includes evaporation, condensation, and precipitation. Water evaporates from oceans, forms clouds through condensation, and then falls back as rain, which is precipitation."

**Response B:** "Imagine a big puddle on a sunny day! The sun heats the water, and it turns into an invisible gas called water vapor, floating up into the sky. This is called **evaporation**. High up, where it's colder, the water vapor turns back into tiny liquid water droplets, forming clouds. This is **condensation**. When the clouds get too full of water, the water falls back down as rain, snow, or hail. This is **precipitation**. The water collects in rivers and oceans, and the whole adventure starts again!"

**Initial Explanation:** "Response B is better because it's more engaging."

**Your Task:** Rewrite the initial explanation to demonstrate a chain-of-thought process.

Improving a Preference Labeling Prompt with Chain-of-Thought

A research team is creating instructions for human labelers who will be rating the quality of two different AI-generated responses to a user's query. The team wants to include an example in their instructions that not only shows a preference but also models a clear, step-by-step reasoning process to guide the labelers. Which of the following examples best accomplishes this goal?

Learn Before

Related