An engineer is training a small 'student' model by learning from a larger 'teacher' model. The training objective is to find the student parameters (θ) that maximize a combined score, formulated as: $\text{score} = \sum (\text{Term A} - \lambda \cdot \text{Term B})$ where 'Term A' measures how well the student predicts the correct, ground-truth answers, and 'Term B' measures how closely the student's outputs match the teacher's outputs. After training, the engineer notices the student model

Analyzing the Knowledge Distillation Hyperparameter

A machine learning team is using a combined objective to train a small 'student' model. The goal is to find the student model's parameters (θ) that maximize the following expression: $\tilde{\theta} = \arg \max_{\theta} \sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} \log \Pr_{\theta}^{s}(\mathbf{y}|\mathbf{x}) - \lambda \cdot \text{Loss}_{\text{kd}}$ The first term, $\log \Pr_{\theta}^{s}(\mathbf{y}|\mathbf{x})$, measures how well the student predicts the ground-truth labels $(\mathbf{y})$.