Applying Masking Strategies for Language Models
You are preparing training data for a language model using the sentence: 'The powerful engine roared to life on the racetrack.'
- Rewrite the sentence, replacing exactly one word with a
[MASK]token to create a training example. - Rewrite the sentence again, this time replacing two different words with
[MASK]tokens to create a second, more challenging training example. - For both of your created examples, briefly describe what the model is expected to predict.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is being trained on the sentence: 'The chef carefully seasoned the delicious soup with a pinch of salt.' Consider two different training examples created from this sentence:
... seasoned the delicious soup with a [MASK] of salt.The [MASK] carefully seasoned the delicious [MASK] with a pinch of salt.
Which statement best analyzes the difference in the learning objective between these two examples?
Applying Masking Strategies for Language Models
Applying Masked Language Modeling