Learn Before
Visual Example of Generator Operation in Replaced Token Detection
To illustrate the generator's role in replaced token detection, an original sequence is first corrupted by masking certain tokens, which are then replaced by the predictions of a small masked language model (the generator). For instance:
In this example, the masked token "hours" is incorrectly predicted and replaced by "decades", whereas "toys" is correctly predicted and replaced by the original token.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
In a two-model pre-training setup, a small 'generator' model first processes an input sentence by masking some words and then filling those masked positions with its own predictions. The resulting, potentially altered, sentence is then passed to a larger 'discriminator' model. What is the most critical function of the generator's output in this process?
Evaluating Corrupted Text for Model Training
A small masked language model is used to create a corrupted version of an input text sequence for a subsequent training task. Arrange the steps this model takes to generate the final corrupted sequence in the correct chronological order.
Visual Example of Generator Operation in Replaced Token Detection
Learn After
A small masked language model, acting as a generator, is given an input sequence where some tokens have been replaced with a [MASK] token: 'The quick [MASK] fox jumps over the [MASK] dog.' The original, unmasked sentence was: 'The quick brown fox jumps over the lazy dog.' Based on the typical behavior of such a generator, which of the following is the most plausible output sequence it would produce to be passed on for further processing?
Analyzing Generator Output in Text Corruption
The generator's role in replaced token detection involves several steps to create a plausible but altered version of an input sentence. Arrange the following events into the correct chronological sequence.