KL Divergence Loss for Knowledge Distillation

A compact computational model is being trained to replicate the probabilistic outputs of a large, established reference model. The training process aims to minimize the dissimilarity between the two models' full output distributions for any given input. Below are the output probability distributions from the reference model and three potential outputs from the compact model for the same input.

Reference Model Output: [0.70, 0.20, 0.10]

Which of the compact model outputs below demonstrates the most successful replication of the reference model's output distribution, considering the goal is to match the entire distribution, not just the most likely outcome?

Compact Model - Output A: [0.65, 0.22, 0.13] Compact Model - Output B: [0.70, 0.10, 0.20] Compact Model - Output C: [0.50, 0.30, 0.20]

Rationale for Distribution Matching in Model Training

Knowledge Distillation Loss using KL Divergence

Analyzing Model Training Scenarios

KL Divergence Loss for Knowledge Distillation

Cross-Entropy Loss for Knowledge Distillation

A large, complex language model is used to generate target probabilities for training a smaller, more efficient model. For the input sentence 'The cat sat on the ___', the large model could produce different probability distributions for the next word. Which of the following distributions, representing $P( ext{output} | ext{context})$, would provide the most informative and nuanced training signal for the smaller model?

Value of the Teacher's Probability Distribution

In a knowledge distillation process for a machine translation task, a large 'teacher' model translates the sentence 'Je suis content' from French to English. Instead of just outputting 'I am happy', the teacher model produces a full probability distribution over the entire English vocabulary for the next words. Which statement best analyzes the significance of this probability distribution ($P_t$) for training the smaller 'student' model?