KL Divergence Loss for Knowledge Distillation

A compact computational model is being trained to replicate the probabilistic outputs of a large, established reference model. The training process aims to minimize the dissimilarity between the two models' full output distributions for any given input. Below are the output probability distributions from the reference model and three potential outputs from the compact model for the same input.

Reference Model Output: [0.70, 0.20, 0.10]

Which of the compact model outputs below demonstrate

Rationale for Distribution Matching in Model Training

Knowledge Distillation Loss using KL Divergence

Analyzing Model Training Scenarios