An engineer is training a small 'student' model by learning from a larger 'teacher' model. The training objective is to find the student parameters (θ) that maximize a combined score, formulated as: $\text{score} = \sum (\text{Term A} - \lambda \cdot \text{Term B})$ where 'Term A' measures how well the student predicts the correct, ground-truth answers, and 'Term B' measures how closely the student's outputs match the teacher's outputs. After training, the engineer notices the student model is replicating systematic errors present in the teacher model, leading to poor performance on a validation set. Which adjustment to the hyperparameter λ is the most appropriate first step to address this issue?

Analyzing the Knowledge Distillation Hyperparameter

A machine learning team is using a combined objective to train a small 'student' model. The goal is to find the student model's parameters (θ) that maximize the following expression: $\tilde{\theta} = \arg \max_{\theta} \sum_{(\mathbf{x}, \mathbf{y}) \in \mathcal{D}} \log \Pr_{\theta}^{s}(\mathbf{y}|\mathbf{x}) - \lambda \cdot \text{Loss}_{\text{kd}}$ The first term, $\log \Pr_{\theta}^{s}(\mathbf{y}|\mathbf{x})$, measures how well the student predicts the ground-truth labels $(\mathbf{y})$. The second term, $\text{Loss}_{\text{kd}}$, measures the difference between the student's and a larger 'teacher' model's predictions. The team is working with a dataset where the ground-truth labels are known to be somewhat noisy and contain occasional errors. However, the large teacher model has been shown to provide very reliable and well-generalized predictions. Given this situation, how should the team adjust the hyperparameter $\lambda$ to optimize the student model's performance?