Learn Before
Learning Contextual Representations via Masked Token Prediction
The primary objective of the masked token prediction task is to train a model to develop a deep understanding of language by learning to generate meaningful representations for tokens based on the surrounding words that form their context.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Comparison of Arbitrary Order Prediction and Masked Language Modeling
Permuted Language Modeling (PLM)
Next Sentence Prediction as an Auxiliary Training Objective
Permuted Language Modeling
Learning Contextual Representations via Masked Token Prediction
A language model is being trained with the following objective: It is given a sentence with a single word randomly obscured, such as 'The quick brown [HIDDEN] jumps over the lazy dog.' The model's only task is to predict the original hidden word, 'fox'. Which of the following best describes the specific contextual information the model is designed to use to make this prediction?
Analyzing a Model Training Process
A language model is being trained on the sentence: 'The quick brown fox jumps over the lazy dog.' Which of the following training scenarios best exemplifies the process of learning by predicting an obscured word using its full surrounding context?
MASS-style Masked Language Modeling
BERT-style Masked Language Modeling
Learn After
Interpreting Contextual Representations
A language model is trained on the following sentence where one word is hidden: 'The scientist meticulously calibrated the [MASK] before the experiment.' The model's primary training objective is to predict the hidden word. Why is this task effective for teaching the model to understand the relationships between words?
A language model is trained using an objective where it must predict randomly hidden words in a sentence based on the surrounding words. After this training, the model will generate the exact same numerical representation for the word 'bank' in the sentences 'He sat on the river bank' and 'She withdrew money from the bank'.