In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

University of California, Berkeley

There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

Types of Adversarial Distillation 



Adversarial Generator is trained to generate synthetic data, which is either added to or used as the training set. 
The distillation loss is $L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$

- $F_t(.) and F_s(.)$ are teacher and student model outputs
- G(z) is the training samples generated by generator G given random input vector z
- $L_G$ is a distillation loss to force the match between predicted vs ground truth probability distributions, e.g. cross entropy or Kullback-Leibler divergence loss. 

Adversarial Generator



A discriminator distinguishes the samples from student and teacher models using either logits or features. A representative model is $L_{GANKD} = L_{CE}(G(F_s(x)), y) + \alphaL_{KL}(G(F_s(x)), F_t(x)) + \betaL_{GAN|(F_s(x), F_t(x))$
- G is a student network
- $L_{GAN}(.)$ is a typical loss function in GAN to make student and teacher outputs similar 


Learn Before

Related