In discriminator-based adversarial distillation, a discriminator distinguishes the samples or features of the student model from those of the teacher model. A representative objective function is:

$$L_{\text{GANKD}} = L_{\text{CE}}(G(F_s(x)), y) + \alpha L_{\text{KL}}(G(F_s(x)), F_t(x)) + \beta L_{\text{GAN}}(F_s(x), F_t(x))$$

Where:
- $$G$$ is a student network.
- $$L_{\text{GAN}}(\cdot)$$ is a typical adversarial loss function used to make the student and teacher outputs similar.

University of California, Berkeley

Google

There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

Types of Adversarial Distillation 


In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

Online Adversarial Knowledge Distillation 



In adversarial distillation, an adversarial generator is trained to generate synthetic data, which is either added to or used as the training set. The distillation loss is:

$$L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$$

Where:
- $$F_t(\cdot)$$ and $$F_s(\cdot)$$ are the teacher and student model outputs.
- $$G(z)$$ represents the training samples generated by generator $$G$$ given a random input vector $$z$$.
- $$L_G$$ is a distillation loss used to force the match between predicted and ground truth probability distributions (e.g., cross-entropy or Kullback-Leibler divergence loss).

Learn Before

Related