Weak-to-Strong Fine-Tuning as a Knowledge Distillation Problem

A research team is adapting a large, powerful language model (the 'strong model') for a specialized task. They lack a large set of human-verified labels, but they have a smaller, less accurate model (the 'weak model') that can generate plausible, albeit imperfect, labels. The team's strategy is to use the weak model to label a large unlabeled dataset and then fine-tune the strong model to mimic the weak model's labeling behavior on this dataset. Which of the following mathematical objectives bes

Analyzing Overfitting in Weak-to-Strong Fine-Tuning

Deconstructing the Weak-to-Strong Fine-Tuning Objective