Based on the evaluation instructions provided in the case study, which response (A or B) is better? Justify your choice by explaining how your selected response better meets the specified criteria.

Google

A prompt for an AI-based preference labeling task in customer service typically establishes a scenario, outlines criteria for a high-quality reply, and provides the text to be evaluated. For example, the prompt might instruct the LLM to choose between two responses based on being courteous, clear, and concise. It then presents the specific customer request (e.g., asking about a delayed order) alongside two options—such as an empathetic apology for the delay (Response A) and a dismissive, unprofessional reply (Response B). The AI is expected to review the criteria and output the preference label, such as "Response A is preferred."

Example Prompt for Evaluating Customer Service Responses

A common customer service scenario involves a polite inquiry about a delayed order. For example, a customer might write, 'Hello, I noticed that my order hasn’t arrived yet, though it was scheduled to arrive several days ago. Could you please update me on its status? Thank you!'. This message clearly states the issue and the requested action, serving as the input for which responses will be generated and evaluated.

Example of a Customer Inquiry about a Delayed Order

A team is creating a set of instructions for human evaluators who will be rating AI-generated customer service replies. The goal is for the evaluators to choose the better of two possible replies to a customer's message. Which of the following instructions would be most effective for this task?

A company is using human evaluators to rate the quality of two different AI-generated responses to a customer's message. Analyze the instructions provided to the evaluators below. Identify two significant weaknesses in these instructions and propose specific revisions to address them. Justify why your proposed changes would lead to more reliable and useful evaluations.

Learn Before

Related