A language model is trained using an objective where every token in the input sentence is replaced by a [MASK] token. The model is then required to reconstruct the entire original sentence. How does the primary skill developed by this training method differ from a method where only a small fraction (e.g., 15%) of the tokens are masked?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Training the Decoder as a Language Model in 100% Masking Scenarios
A language model is trained using an objective where every token in the input sentence is replaced by a
[MASK]token. The model is then required to reconstruct the entire original sentence. How does the primary skill developed by this training method differ from a method where only a small fraction (e.g., 15%) of the tokens are masked?Constructing a 100% Masked Training Example
Evaluating a Model Training Strategy