Evaluating a Text Generation Strategy
A software team is using a generative model to summarize news articles. To improve the quality of the summaries, they implement a new process: for each article, they generate 10 candidate summaries. Then, they select the final summary by choosing the candidate that has the highest average similarity score when compared against all other candidates in the set. Analyze this approach. Why is this method of selecting the 'most representative' summary expected to yield a higher-quality result than simply using the first summary the model generates?
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Risk Function in Minimum Bayes Risk Decoding
Risk of an Output in Minimum Bayes Risk Decoding
A language model is prompted to solve the math problem 'What is 7 + 8?'. To improve reliability, the model generates five different outputs using a sampling strategy: [15, 14, 15, 15, 16]. A selection process is then used to choose the final answer by identifying the candidate that minimizes the expected disagreement with the other generated candidates. Which output will be selected?
Connecting Self-Consistency to a Formal Framework
Evaluating a Text Generation Strategy