Learn Before
GAN-based Training for Replaced Token Detection
An alternative to the standard joint training in Replaced Token Detection is to use a Generative Adversarial Network (GAN) framework. In this setup, the generator's objective shifts from simple prediction to actively trying to fool the discriminator. The discriminator, in turn, is trained to distinguish between the generator's output and the original data distribution. However, this adversarial approach tends to complicate the training process and is generally more difficult to scale effectively for this task.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
GAN-based Training for Replaced Token Detection
In a language model pre-training setup, a 'generator' network corrupts an input sentence by replacing some tokens. A separate 'discriminator' network is then tasked with identifying which tokens in the corrupted sentence are original and which are replacements. If both networks are trained simultaneously, which statement best distinguishes their respective optimization goals?
Differentiating Training Objectives in a Two-Network Model
Analysis of Joint Training Dynamics in a Two-Network Model