Visual Example of Discriminator Operation in Replaced Token Detection

In a particular self-supervised learning setup, a 'generator' model first processes an input sentence and replaces some of its words with plausible alternatives. A second, more powerful 'discriminator' model then receives this altered sentence. The discriminator's task is to examine each word and determine if it is identical to the word in the original, unaltered sentence.

Consider this example:

• Original Sentence: "The scientist discovered the new element."
• Altered Sentence from Generator: "The scientist found the new element."

Given the discriminator's task, how should it classify the words 'found' and 'element' from the altered sentence?

In a replaced token detection task, a generator model is given the sentence 'The chef cooked the meal' with the word 'cooked' masked. The generator predicts the word 'cooked' to fill the mask. The discriminator then receives the sentence 'The chef cooked the meal'. According to the discriminator's objective, it should classify the word 'cooked' as 'replaced'.

Discriminator Performance Analysis