KL Divergence Loss for Knowledge Distillation

Cross-Entropy Loss for Knowledge Distillation

A large, complex language model is used to generate target probabilities for training a smaller, more efficient model. For the input sentence 'The cat sat on the ___', the large model could produce different probability distributions for the next word. Which of the following distributions, representing $P( ext{output} | ext{context})$, would provide the most informative and nuanced training signal for the smaller model?

Value of the Teacher's Probability Distribution

In a knowledge distillation process for a machine translation task, a large 'teacher' model translates the sentence 'Je suis content' from French to English. Instead of just outputting 'I am happy', the teacher model produces a full probability distribution over the entire English vocabulary for the next words. Which statement best analyzes the significance of this probability distribution ($P_t$) for training the smaller 'student' model?