Google

An extreme application of Masked Language Modeling (MLM) involves masking 100% of the tokens in a sequence, which effectively transforms the training objective into a sequence generation task. In this scenario, an input consisting of a `[CLS]` token followed by `[MASK]` tokens for every word is used to train the model to generate the complete, original sentence. For example: `[CLS] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] → ⟨s⟩ The puppies are frolicking outside the house .`

Example of Full Sequence Generation via 100% Masking

In the specific case of Masked Language Modeling where 100% of the input tokens are masked, the training objective becomes equivalent to sequence generation. Consequently, the decoder is trained to operate as a language model, responsible for generating the entire original text.

Training the Decoder as a Language Model in 100% Masking Scenarios

Consider the sentence: 'The quick brown fox jumps.' For a model being trained with an objective where 100% of the input tokens are masked, what would the input sequence fed to the model look like, and what would be the corresponding target output sequence the model is trained to generate? Assume the model uses `[CLS]` as a starting token and `[MASK]` for masked tokens.

Constructing a 100% Masked Training Example

Based on this training setup, what specific capability is the model being optimized to learn, and why is this method particularly suitable for the team's goal?

Evaluating a Model Training Strategy

A language model is trained using an objective where every token in the input sentence is replaced by a `[MASK]` token. The model is then required to reconstruct the entire original sentence. How does the primary skill developed by this training method differ from a method where only a small fraction (e.g., 15%) of the tokens are masked?

Learn Before

Related