Adversarial learning is good for generative networks. The discriminator in a GAN estimates the probability that a sample is from the training distribution while the generator tries to trick the discriminator using generated data samples. This allows the teacher and student networks to better understand the true data distribution. 

Adversarial Distillation



Multi-Teacher Distillation 



Transferring knowledge between modalities is important because data/labels for every modality are not available during training/testing. An example is transferring the knowledge of a teacher model pre-tained on one modality to the student with a new unlabeled input modality, using pair-wise sample registration. Features obtained from the teacher are used for supervised training of the student. This performs well in visual recognition tasks in cross-modal scenarios. 

Cross-Model Distillation



Graphs are used to explore intra-data relationships and goes with relation-based knowledge. Graph based distillation uses graphs as either the carrier of teacher knowledge or to control the passing of the teacher knowledge. It can transfer the structure knowledge of data, but constructing the graph to model this knowledge is a challenge. 

Graph-Based Distillation



Attention reflects the neuron activations of convolutional neural networks, and different attention transfer mechanisms are defined for distilling knowledge from the teacher network. The main task is defining the attention maps for feature embedding in neural network layers, because embedding knowledge is transferred using attention map functions. 

Attention-Based Distillation


To solve the problem of unavailable data, this method uses no training data, using synthetically generated data, which can be retrieved using a GAN, e.g. data can be reconstructed using layer activations of the teacher network. However, the task of generating data is challenging.

Data-Free Distillation



Network quantization reduces a neural network’s computational complexity by converting high-precision networks to low precision. A high-precision teacher network, quantized on feature maps, distills knowledge to a low-precision network. This has been recently used with self-distillation training schemes, where the teacher and student parameters are the same.

Quantized Distillation



Lifelong learning, which includes continual, continuous, and meta learning, accumulates and transfers the previously learned knowledge  into future learning. KD does this effectively without catastrophic forgetting, and meta-learning has been used to determine what and where to make the transfer. Global distillation, knowledge distillation-based lifelong GAN, multi-model distillation, and other KD methods also address the problem of catastrophic forgetting. 

Lifelong Distillation



Neural Architecture Search is a popular auto machine learning technique which automatically identifies deep neural models and adaptively learns deep neural structures. NAS finds the appropriate student architecture in oracle-based and architecture-aware KD. This helps the student to learn well from the teacher despite the capacity gap between the two models. Feature transfer is effectively supervised by the teacher, and KD also improves the efficiency of NAS. 


NAS-Based Distillation




Directly matching the response-based knowledge, feature-based knowledge, or representation distributions in feature space between the the teacher and student models is a straightforward method for knowledge transfer. There are algorithms for different types of distillation methods for knowledge transfer. 

University of California, Berkeley

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Teacher-student architecture is a carrier to form the knowledge transfer. Focused on the relationship between the teacher and student model structure, it consists of selecting and designing proper structures of teacher and student. 


Learn Before

Related

Learn After