Evaluating Masking Strategies for Model Training
A researcher wants to train a language model to specifically learn the relationship between a subject (who performs an action) and the verb (the action itself). Based on the two training strategies presented below, which one is more suitable for this goal? Justify your evaluation.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Evaluation in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Evaluating Masking Strategies for Model Training
A language model is being trained using a masked language modeling objective. If the original input sentence is 'The innovative chef prepares a delicious meal for the guests.', which of the following masked versions would be the most difficult for the model to reconstruct accurately?
In a masked language modeling task with the input
The student opened their [M] to study for the exam., the model's prediction for the masked word is solely determined by the words that come before the mask (The student opened their).