Optimizing a Distillation Training Process
Based on the scenario provided, propose a more computationally efficient training strategy that still leverages the teacher model's knowledge but avoids the problematic calculation. Describe the step-by-step data generation and training flow for your proposed strategy.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A team is developing a small, efficient text-generation model (the 'student') by training it to imitate a much larger, powerful model (the 'teacher'). Their training method requires the student to learn from the full probability distribution the teacher assigns over all possible next words. They discover this is computationally infeasible because their vocabulary contains hundreds of thousands of words, and calculating the training objective for a single example requires summing over this entire vocabulary. Which of the following strategies provides the most practical solution to this specific computational problem while still using the teacher's guidance?
Evaluating a Distillation Training Strategy
Optimizing a Distillation Training Process