Online distillation is a one-phase, end-to-end training scheme. However, existing methods do not address the high-capacity teacher in online settings. 


Pros and Cons


- In deep mutual learning, multiple neural networks work collaboratively. Any network can be the teacher or student model during training. 
- Co-distillation trains multiple models in parallel with the same architecture. Any model is trained by transferring knowledge from the other models.

Varieties of Online Distillation


In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


University of California, Berkeley

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

An Introduction to Deep Reinforcement Learning
https://arxiv.org/pdf/1811.12560.pdf

Reference for Deep Reinforcement Learning

Offline methods assume the teacher knowledge/model is already defined, and focus on improving knowledge transfer, such as design of knowledge and loss functions for feature or distribution matching. They usually have one-way knowledge transfer and two-phase training procedure. 


Offline Distillation

Online Distillation


Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


Learn Before

Related

Learn After