In adversarial distillation, an adversarial generator is trained to generate synthetic data, which is either added to or used as the training set. The distillation loss is:

$$L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$$

Where:
- $$F_t(\cdot)$$ and $$F_s(\cdot)$$ are the teacher and student model outputs.
- $$G(z)$$ represents the training samples generated by generator $$G$$ given a random input vector $$z$$.
- $$L_G$$ is a distillation loss used to force the match between predicted and ground truth probability distributions (e.g., cross-entropy or Kullback-Leibler divergence loss).

University of California, Berkeley

Google

There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

Types of Adversarial Distillation 



A discriminator distinguishes the samples from student and teacher models using either logits or features. A representative model is $L_{GANKD} = L_{CE}(G(F_s(x)), y) + \alphaL_{KL}(G(F_s(x)), F_t(x)) + \betaL_{GAN|(F_s(x), F_t(x))$
- G is a student network
- $L_{GAN}(.)$ is a typical loss function in GAN to make student and teacher outputs similar 


Discriminator


In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

Learn Before

Related