A machine learning team is developing a compact language model (the 'student') by training it to learn from a much larger, high-performing model (the 'teacher'). They conduct two experiments with identical student model architectures:

• Experiment 1: The student model is trained solely by minimizing the difference between its final output predictions and the teacher model's final output predictions.
• Experiment 2: In addition to matching the final predictions, the student model is

When using a large 'teacher' model to train a smaller 'student' model, the only way to transfer knowledge is by training the student to replicate the teacher's final output predictions.

Enhancing Knowledge Transfer in Model Distillation