KL Divergence Loss for Knowledge Distillation

Cross-Entropy Loss for Knowledge Distillation

A large, complex language model is used to generate target probabilities for training a smaller, more efficient model. For the input sentence 'The cat sat on the ___', the large model could produce different probability distributions for the next word. Which of the following distributions, representing $P( ext{output} | ext{context})$, would provide the most informative and nuanced training signal for the smaller model?

Value of the Teacher's Probability Distribution

In a knowledge distillation process for a machine translation task, a large 'teacher' model translates the sentence 'Je suis content' from French to English. Instead of just outputting 'I am happy', the teacher model produces a full probability distribution over the entire English vocabulary for the next words. Which statement best analyzes the significance of this probability distribution ($P_t$) for training the smaller 'student' model?

General Loss Function for Knowledge Distillation

Loss Function for Conditional Probability Distributions (Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}))

A machine learning team is developing a compact, efficient language model, which we'll call model 's'. The model's behavior is governed by a set of tunable weights, denoted by θ. For a given task, the model receives a simplified context input, c', and a latent variable, z, and then generates a probability distribution over all possible outputs. Which of the following expressions correctly represents this model's output probability distribution?

In the expression $P_\theta^s = \text{Pr}_\theta^s(\cdot|\mathbf{c}', \mathbf{z})$, which describes a model's output probability distribution, match each symbol to its correct description.

Applying the Student Model Probability Notation

General Loss Function for Knowledge Distillation

Loss Function for Conditional Probability Distributions (Loss(\\text{Pr}^t(\\cdot|\\cdot), \\text{Pr}_\\theta^s(\\cdot|\\cdot), \\mathbf{x}))