Training with Teacher-Generated Outputs as a Distillation Variant
To circumvent the computational challenge of summing over all possible outputs, a common variant in knowledge distillation is to train the student model using specific outputs generated by the teacher model. For each training sample, the teacher model produces an output, which then serves as the target for training the student model, avoiding the need to iterate through the entire output space.
0
1
Tags
Ch.3 Prompting - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Learn After
A team is developing a small, efficient text-generation model (the 'student') by training it to imitate a much larger, powerful model (the 'teacher'). Their training method requires the student to learn from the full probability distribution the teacher assigns over all possible next words. They discover this is computationally infeasible because their vocabulary contains hundreds of thousands of words, and calculating the training objective for a single example requires summing over this entire vocabulary. Which of the following strategies provides the most practical solution to this specific computational problem while still using the teacher's guidance?
Evaluating a Distillation Training Strategy
Optimizing a Distillation Training Process