
Adversarial Generator is trained to generate synthetic data, which is either added to or used as the training set. 
The distillation loss is $L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$

- $F_t(.) and F_s(.)$ are teacher and student model outputs
- G(z) is the training samples generated by generator G given random input vector z
- $L_G$ is a distillation loss to force the match between predicted vs ground truth probability distributions, e.g. cross entropy or Kullback-Leibler divergence loss. 

University of California, Berkeley

There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

Types of Adversarial Distillation 


Adversarial Generator



A discriminator distinguishes the samples from student and teacher models using either logits or features. A representative model is $L_{GANKD} = L_{CE}(G(F_s(x)), y) + \alphaL_{KL}(G(F_s(x)), F_t(x)) + \betaL_{GAN|(F_s(x), F_t(x))$
- G is a student network
- $L_{GAN}(.)$ is a typical loss function in GAN to make student and teacher outputs similar 


Discriminator


In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

Learn Before

Related