Evaluating Candidate Sets for Selection
A system is designed to generate multiple candidate responses to a prompt and then select the best one. For the prompt 'Describe the primary function of a car's engine,' the system generates the two sets of candidates below.
Set A:
- The engine's main job is to convert fuel into motion.
- The principal role of an engine is transforming fuel into mechanical energy.
- An engine's primary purpose is to turn chemical energy from fuel into movement.
Set B:
- The engine combusts fuel to power the wheels, making the car move.
- The engine also powers the alternator, which charges the battery and runs the car's electrical systems.
- The engine's cooling system, including the radiator, must be maintained to prevent overheating.
Which set of candidates, A or B, provides a better basis for the selection system to choose a high-quality, comprehensive response? Justify your reasoning by comparing the characteristics of the two sets.
0
1
Tags
Ch.5 Inference - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Strategies to Enhance Output Diversity for Reranking
Balancing Candidate Quality and Diversity in Reranking
An engineering team implements a system to improve a language model's output. For each user query, the system generates 10 candidate responses and then uses a highly accurate reward model to select the best one. Despite the high accuracy of the reward model, the team observes that the final selected response is rarely a significant improvement over any of the other 9 candidates. Which of the following is the most likely underlying cause for this lack of significant improvement?
Diagnosing Reranking System Performance
Evaluating Candidate Sets for Selection
Critique of Reranking Effectiveness