To build the student model in context distillation, a new dataset, denoted as $$\mathcal{D}'$$, is constructed. Each sample in this dataset is a tuple $$\mathbf{x}' = (\mathbf{c}, \mathbf{c}', \mathbf{z})$$, which consists of an original detailed instruction $$\mathbf{c}$$, a corresponding simplified instruction $$\mathbf{c}'$$, and a user input $$\mathbf{z}$$.

Google

Context distillation is a knowledge distillation method designed to adapt large language models (LLMs) to follow simplified instructions. It involves training a student model to make predictions based on user inputs and simplified contexts (such as condensed instructions). This is achieved by transferring knowledge from a well-trained, instruction-following teacher model that processes the original, detailed instructions. The student model learns by minimizing the loss between its predictions and those produced by the teacher model.

Context Distillation

Reference of Foundations of Large Language Models Course

In context distillation, building the teacher model follows a standard fine-tuning process. This involves collecting a dataset comprising instructions, user inputs, and correct responses, and then continuing to train a pre-trained language model with this dataset so it becomes proficient at following detailed instructions.

Teacher Model in Context Distillation

Student Model Dataset in Context Distillation

Knowledge distillation in the context distillation method is performed by minimizing a loss function defined on the outputs of the teacher and student models:

$$\hat{\theta} = \argmin_{\theta} \sum_{\mathbf{x}' \in \mathcal{D}'} \mathrm{Loss}(\mathrm{Pr}^{t}(\cdot|\cdot),\ \mathrm{Pr}_{\theta}^{s}(\cdot|\cdot),\ \mathbf{x}')$$

where $$\mathrm{Pr}^{t}(\cdot|\cdot)$$ denotes the pre-trained teacher model, and $$\mathrm{Pr}_{\theta}^{s}(\cdot|\cdot)$$ denotes the student model with the parameters $$\theta$$.

Learn Before

Related