Weak-to-Strong Fine-Tuning as a Knowledge Distillation Problem

A research team is adapting a large, powerful language model (the 'strong model') for a specialized task. They lack a large set of human-verified labels, but they have a smaller, less accurate model (the 'weak model') that can generate plausible, albeit imperfect, labels. The team's strategy is to use the weak model to label a large unlabeled dataset and then fine-tune the strong model to mimic the weak model's labeling behavior on this dataset. Which of the following mathematical objectives best represents the goal of finding the optimal strong model parameters, $\tilde{\theta}$, that maximize the strong model's ability to predict the labels, $\hat{\mathbf{y}}$, generated by the weak model for a given set of inputs, $\mathbf{x}$?

Analyzing Overfitting in Weak-to-Strong Fine-Tuning

Deconstructing the Weak-to-Strong Fine-Tuning Objective