Online distillation is a one-phase, end-to-end training scheme. However, existing methods do not address the high-capacity teacher in online settings. 


University of California, Berkeley

In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


Online Distillation


Pros and Cons


- In deep mutual learning, multiple neural networks work collaboratively. Any network can be the teacher or student model during training. 
- Co-distillation trains multiple models in parallel with the same architecture. Any model is trained by transferring knowledge from the other models.

Learn Before

Related