University of California, Berkeley

Directly matching the response-based knowledge, feature-based knowledge, or relation-based knowledge between the teacher and student models is a common way to transfer knowledge in knowledge distillation (KD). Different algorithms specialize in different mechanisms for this transfer, including: - Adversarial Distillation - Multi-Teacher Distillation - Cross-Model Distillation - Graph-Based Distillation - Attention-Based Distillation - NAS-Based Distillation - Quantized Distillation - Lifelong Distillation - Data-Free Distillation

Distillation Algorithms

Multi-Teacher Distillation 



Attention reflects the neuron activations of convolutional neural networks, and different attention transfer mechanisms are defined for distilling knowledge from the teacher network. The main task is defining the attention maps for feature embedding in neural network layers, because embedding knowledge is transferred using attention map functions. 

Attention-Based Distillation


Network quantization reduces a neural network’s computational complexity by converting high-precision networks to low precision. A high-precision teacher network, quantized on feature maps, distills knowledge to a low-precision network. This has been recently used with self-distillation training schemes, where the teacher and student parameters are the same.

Quantized Distillation



Lifelong learning, which includes continual, continuous, and meta learning, accumulates and transfers the previously learned knowledge  into future learning. KD does this effectively without catastrophic forgetting, and meta-learning has been used to determine what and where to make the transfer. Global distillation, knowledge distillation-based lifelong GAN, multi-model distillation, and other KD methods also address the problem of catastrophic forgetting. 

Lifelong Distillation



Neural Architecture Search is a popular auto machine learning technique which automatically identifies deep neural models and adaptively learns deep neural structures. NAS finds the appropriate student architecture in oracle-based and architecture-aware KD. This helps the student to learn well from the teacher despite the capacity gap between the two models. Feature transfer is effectively supervised by the teacher, and KD also improves the efficiency of NAS. 


NAS-Based Distillation



Adversarial distillation is a knowledge distillation algorithm that incorporates adversarial learning concepts, such as those used in Generative Adversarial Networks (GANs), to enhance knowledge transfer. This approach typically involves using a generator to create synthetic training data, or employing a discriminator to distinguish between the outputs or feature distributions of the teacher and student networks, compelling the student to more closely mimic the teacher.

Adversarial Distillation

Transferring knowledge between modalities is important because data/labels for every modality are not available during training/testing. An example is transferring the knowledge of a teacher model pre-trained on one modality to the student with a new unlabeled input modality, using pair-wise sample registration. Features obtained from the teacher are used for supervised training of the student. This performs well in visual recognition tasks in cross-modal scenarios.

Cross-Model Distillation

Graph-based distillation uses graphs to explore intra-data relationships, aligning with relation-based knowledge transfer. In this approach, graphs act either as the carrier of the teacher's knowledge or as a mechanism to control its transmission to the student model. While it effectively transfers the structural knowledge of data, constructing an accurate graph to model this knowledge remains a significant challenge.

Graph-Based Distillation

To solve the problem of unavailable data, data-free distillation uses no training data. Instead, it relies on synthetically generated data, which can be produced using a Generative Adversarial Network (GAN). For example, data can be reconstructed using the layer activations of the teacher network. However, the task of generating this synthetic data is challenging.

Learn Before

Related