Offline methods are simple and easy to implement. While it cannot avoid the high-capacity complex teacher with huge training time, student models usually train efficiently. 


Pros and Cons 

Offline methods assume the teacher knowledge/model is already defined, and focus on improving knowledge transfer, such as design of knowledge and loss functions for feature or distribution matching. They usually have one-way knowledge transfer and two-phase training procedure. 


University of California, Berkeley

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Offline Distillation

In online distillation, the teacher and student models are updated simultaneously. It seeks to improve the student performance when a large-capacity high performance teacher model is not available. 


Online Distillation


Self-distillation is a special case of online distillation where the teacher and student use the same networks. Essentially, the student learns knowledge on its own.  


Learn Before

Related

Learn After