
Adversarial Generator is trained to generate synthetic data, which is either added to or used as the training set. 
The distillation loss is $L_{KD} = L_G(F_t(G(z)), F_s(G(z)))$

- $F_t(.) and F_s(.)$ are teacher and student model outputs
- G(z) is the training samples generated by generator G given random input vector z
- $L_G$ is a distillation loss to force the match between predicted vs ground truth probability distributions, e.g. cross entropy or Kullback-Leibler divergence loss. 

Adversarial Generator



A discriminator distinguishes the samples from student and teacher models using either logits or features. A representative model is $L_{GANKD} = L_{CE}(G(F_s(x)), y) + \alphaL_{KL}(G(F_s(x)), F_t(x)) + \betaL_{GAN|(F_s(x), F_t(x))$
- G is a student network
- $L_{GAN}(.)$ is a typical loss function in GAN to make student and teacher outputs similar 


Discriminator


In an online implementation, the teacher and student are jointly optimized at each iteration. A small GAN student network copies a larger GAN network through knowledge transfer. KD can be used to compress GANs. 

Online Adversarial Knowledge Distillation 



There are 3 categories of adversarial learning-based distillation methods. 

- Adversarial Generator
- Discriminator 
- Online Adversarial Knowledge Distillation 

University of California, Berkeley

Adversarial learning is good for generative networks. The discriminator in a GAN estimates the probability that a sample is from the training distribution while the generator tries to trick the discriminator using generated data samples. This allows the teacher and student networks to better understand the true data distribution. 

Learn Before

Related

Learn After