In generator-based adversarial distillation, an adversarial generator is trained to produce synthetic data that is added to or used as the training set for knowledge distillation. The distillation loss is: $$L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$$ Where: - $$F_t(\cdot)$$ and $$F_s(\cdot)$$ are the teacher and student model outputs. - $$G(z)$$ represents the training samples generated by generator $$G$$ given a random input vector $$z$$. - $$L_G$$ is a distillation loss used to force the match between predicted and ground truth probability distributions (e.g., cross-entropy or Kullback-Leibler divergence loss).

Google

University of California, Berkeley

There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

Types of Adversarial Distillation 


In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

Online Adversarial Knowledge Distillation 



In discriminator-based adversarial distillation, a discriminator distinguishes the samples or features of the student model from those of the teacher model. A representative objective function is:

$$L_{\text{GANKD}} = L_{\text{CE}}(G(F_s(x)), y) + \alpha L_{\text{KL}}(G(F_s(x)), F_t(x)) + \beta L_{\text{GAN}}(F_s(x), F_t(x))$$

Where:
- $$G$$ is a student network.
- $$L_{\text{GAN}}(\cdot)$$ is a typical adversarial loss function used to make the student and teacher outputs similar.

Learn Before

Related