To overcome the computational infeasibility of the sequence-level loss, a variant of context distillation trains the student model using specific outputs generated by the teacher model. For each sample, the teacher model produces an output $\hat{\mathbf{y}} = \argmax_{\mathbf{y}} \log \mathrm{Pr}^{t}(\mathbf{y}|\mathbf{c},\mathbf{z})$, which is then considered the target for learning. The simplified loss function becomes:

$\mathrm{Loss} = \log \mathrm{Pr}_{\theta}^{s}(\hat{\mathbf{y}}|\mathbf{c}', \mathbf{z})$