
Attention reflects the neuron activations of convolutional neural networks, and different attention transfer mechanisms are defined for distilling knowledge from the teacher network. The main task is defining the attention maps for feature embedding in neural network layers, because embedding knowledge is transferred using attention map functions. 

University of California, Berkeley


Directly matching the response-based knowledge, feature-based knowledge, or representation distributions in feature space between the the teacher and student models is a straightforward method for knowledge transfer. There are algorithms for different types of distillation methods for knowledge transfer. 

Distillation Algorithms 


Adversarial learning is good for generative networks. The discriminator in a GAN estimates the probability that a sample is from the training distribution while the generator tries to trick the discriminator using generated data samples. This allows the teacher and student networks to better understand the true data distribution. 

Adversarial Distillation



Multi-Teacher Distillation 



Transferring knowledge between modalities is important because data/labels for every modality are not available during training/testing. An example is transferring the knowledge of a teacher model pre-tained on one modality to the student with a new unlabeled input modality, using pair-wise sample registration. Features obtained from the teacher are used for supervised training of the student. This performs well in visual recognition tasks in cross-modal scenarios. 

Cross-Model Distillation



Graphs are used to explore intra-data relationships and goes with relation-based knowledge. Graph based distillation uses graphs as either the carrier of teacher knowledge or to control the passing of the teacher knowledge. It can transfer the structure knowledge of data, but constructing the graph to model this knowledge is a challenge. 

Graph-Based Distillation


Attention-Based Distillation


To solve the problem of unavailable data, this method uses no training data, using synthetically generated data, which can be retrieved using a GAN, e.g. data can be reconstructed using layer activations of the teacher network. However, the task of generating data is challenging.

Data-Free Distillation



Network quantization reduces a neural network’s computational complexity by converting high-precision networks to low precision. A high-precision teacher network, quantized on feature maps, distills knowledge to a low-precision network. This has been recently used with self-distillation training schemes, where the teacher and student parameters are the same.

Quantized Distillation



Lifelong learning, which includes continual, continuous, and meta learning, accumulates and transfers the previously learned knowledge  into future learning. KD does this effectively without catastrophic forgetting, and meta-learning has been used to determine what and where to make the transfer. Global distillation, knowledge distillation-based lifelong GAN, multi-model distillation, and other KD methods also address the problem of catastrophic forgetting. 

Lifelong Distillation



Neural Architecture Search is a popular auto machine learning technique which automatically identifies deep neural models and adaptively learns deep neural structures. NAS finds the appropriate student architecture in oracle-based and architecture-aware KD. This helps the student to learn well from the teacher despite the capacity gap between the two models. Feature transfer is effectively supervised by the teacher, and KD also improves the efficiency of NAS. 

Learn Before

Related