Learn Before
Critique of a Modified Pre-training Strategy
A data scientist is pre-training a language model using a masked prediction objective. In their setup, 15% of the input tokens are selected for prediction. However, to simplify the process, they decide to replace 100% of these selected tokens with a special [MASK] token. After pre-training, they observe that the model's performance on various downstream fine-tuning tasks is significantly worse than expected. Evaluate the data scientist's simplified pre-training strategy. Why did this modification likely lead to poor performance during fine-tuning, despite the model performing well on the pre-training objective itself?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Example of an Unchanged Token in a BERT Input Sequence
A language model is pre-trained using a method where 15% of the words in an input sentence are selected for prediction. Of these selected words, a small fraction (10%) are intentionally left in their original form, while the model is still tasked with predicting them based on the surrounding context. What is the most significant reason for this strategy of leaving some target words unchanged?
Calculating Token Modifications in Pre-training
Critique of a Modified Pre-training Strategy
Purpose of Unchanged Tokens in BERT's MLM Strategy