Multi-Teacher Distillation 



Attention reflects the neuron activations of convolutional neural networks, and different attention transfer mechanisms are defined for distilling knowledge from the teacher network. The main task is defining the attention maps for feature embedding in neural network layers, because embedding knowledge is transferred using attention map functions. 

Attention-Based Distillation


Network quantization reduces a neural network’s computational complexity by converting high-precision networks to low precision. A high-precision teacher network, quantized on feature maps, distills knowledge to a low-precision network. This has been recently used with self-distillation training schemes, where the teacher and student parameters are the same.

Quantized Distillation



Lifelong learning, which includes continual, continuous, and meta learning, accumulates and transfers the previously learned knowledge  into future learning. KD does this effectively without catastrophic forgetting, and meta-learning has been used to determine what and where to make the transfer. Global distillation, knowledge distillation-based lifelong GAN, multi-model distillation, and other KD methods also address the problem of catastrophic forgetting. 

Lifelong Distillation



Neural Architecture Search is a popular auto machine learning technique which automatically identifies deep neural models and adaptively learns deep neural structures. NAS finds the appropriate student architecture in oracle-based and architecture-aware KD. This helps the student to learn well from the teacher despite the capacity gap between the two models. Feature transfer is effectively supervised by the teacher, and KD also improves the efficiency of NAS. 


NAS-Based Distillation



Adversarial distillation is a knowledge distillation algorithm that incorporates adversarial learning concepts, such as those used in Generative Adversarial Networks (GANs), to enhance knowledge transfer. This approach typically involves using a generator to create synthetic training data, or employing a discriminator to distinguish between the outputs or feature distributions of the teacher and student networks, compelling the student to more closely mimic the teacher.

Adversarial Distillation

Transferring knowledge between modalities is important because data/labels for every modality are not available during training/testing. An example is transferring the knowledge of a teacher model pre-trained on one modality to the student with a new unlabeled input modality, using pair-wise sample registration. Features obtained from the teacher are used for supervised training of the student. This performs well in visual recognition tasks in cross-modal scenarios.

Cross-Model Distillation

Graph-based distillation uses graphs to explore intra-data relationships, aligning with relation-based knowledge transfer. In this approach, graphs act either as the carrier of the teacher's knowledge or as a mechanism to control its transmission to the student model. While it effectively transfers the structural knowledge of data, constructing an accurate graph to model this knowledge remains a significant challenge.

Graph-Based Distillation

To solve the problem of unavailable data, data-free distillation uses no training data. Instead, it relies on synthetically generated data, which can be produced using a Generative Adversarial Network (GAN). For example, data can be reconstructed using the layer activations of the teacher network. However, the task of generating this synthetic data is challenging.

Data-Free Distillation

Directly matching the response-based knowledge, feature-based knowledge, or relation-based knowledge between the teacher and student models is a common way to transfer knowledge in knowledge distillation (KD). Different algorithms specialize in different mechanisms for this transfer, including: - Adversarial Distillation - Multi-Teacher Distillation - Cross-Model Distillation - Graph-Based Distillation - Attention-Based Distillation - NAS-Based Distillation - Quantized Distillation - Lifelong Distillation - Data-Free Distillation

University of California, Berkeley

Google

- Knowledge
- Distillation algorithm
- Teacher-student architecture


Components of a Knowledge Distillation System


There are different types of knowledge in knowledge distillation (KD). They are 
- Response based knowledge
- Feature-based knowledge
- Relation-based knowledge  


Categories of Knowledge 

There are 3 categories of distillation (training) schemes for teacher and student models. They are divided according to whether the teacher and the student model are updated simultaneously or not. 

- Offline Distillation 
- Online Distillation
- Self-Distillation 

Distillation (Training) Schemes 

Distillation Algorithms

In knowledge distillation, the teacher-student architecture is the pair of network structures that serves as the carrier for knowledge transfer. Designing it involves selecting suitable structures for the teacher model and the student model, because the capacity gap between the two models affects how effectively the student can absorb the teacher's knowledge.

Learn Before

Related

Learn After