Evaluating Data Generation Strategies for Model Generalization
A company is developing a customer support chatbot. They plan to fine-tune a large language model using synthetically generated data. They are considering two strategies for creating the input prompts that will be fed to a generator model:
Strategy 1: Use their entire historical log of 500,000 customer support tickets as the input prompts. Strategy 2: Use a set of 20,000 highly varied and unconventional prompts designed by a team of creative writers to mimic unexpected user behavior.
Evaluate these two strategies. In your response, argue which strategy is more likely to result in a chatbot that generalizes well to a wide range of future, real-world user queries. Justify your conclusion by explaining the potential risks associated with the less effective strategy.
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Generating Inputs and Outputs for Comprehensive Fine-Tuning
Chatbot Performance Analysis
A development team is fine-tuning a large language model to act as a technical support chatbot. To create a large training dataset, they use a powerful base model to generate responses to a set of 10,000 technical questions curated by their internal support staff. After deployment, the chatbot excels at answering questions similar to those in the curated set but struggles significantly with novel or unusually phrased queries from real users. Which of the following best analyzes the primary weakness in their data generation strategy?
Evaluating Data Generation Strategies for Model Generalization