Evaluating Generation Strategies in a Q&A System
A developer is building a question-answering system. They have successfully implemented a retrieval component that finds relevant text passages for a user's query. The developer proposes two different approaches for the final generation step:
- Approach A: Ignore the original query and simply ask a language model to summarize the retrieved text passages.
- Approach B: Combine the original query with the retrieved text passages and ask the language model to answer the query based on the provided text.
Critique both approaches. Which approach is more likely to produce a helpful and accurate answer for the user, and why? Justify your reasoning by explaining the role of each input component (the original query and the retrieved text).
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Prompting LLMs with Retrieved Texts in RAG
A system is designed to answer questions using a two-step process: first, it finds relevant documents from a database, and second, it uses a large language model to generate a final answer. A user asks, 'What is the battery life of the new "Innovate X" phone?' The system retrieves the following text: 'The Innovate X phone features a 5000mAh battery, providing up to 48 hours of talk time.' Which of the following inputs to the language model is structured to produce the most accurate and relevant final answer?
Diagnosing a Generation Failure in an Information System
Evaluating Generation Strategies in a Q&A System