Learn Before
Analysis of Joint Training Dynamics in a Two-Network Model
In a pre-training framework where one neural network (the 'generator') replaces tokens in a text and a second network (the 'discriminator') identifies these replacements, both are trained simultaneously by combining their respective loss functions. Analyze the dynamic relationship between the generator's training objective (predicting original tokens) and the discriminator's training objective (classifying tokens). Specifically, explain how the improvement of the generator's performance creates a more challenging and effective learning task for the discriminator.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
GAN-based Training for Replaced Token Detection
In a language model pre-training setup, a 'generator' network corrupts an input sentence by replacing some tokens. A separate 'discriminator' network is then tasked with identifying which tokens in the corrupted sentence are original and which are replacements. If both networks are trained simultaneously, which statement best distinguishes their respective optimization goals?
Differentiating Training Objectives in a Two-Network Model
Analysis of Joint Training Dynamics in a Two-Network Model