Causal Language Modeling as a Special Case of Masked Language Modeling

Example of Masked Language Modeling Prediction

Consider two different approaches for training a language model to predict a specific word within a sentence.

Approach 1: The model is trained to predict the next word in a sequence, using only the words that have appeared before it.

Approach 2: The model is trained to predict a word that has been intentionally hidden, using all the other visible words in the sentence, both those that come before and after the hidden word.

If both models are tasked with predicting the word 'jumps' in the s

Choosing the Right Contextual Approach for Language Tasks

Match each description of a language model's prediction task or characteristic to the type of contextual information it utilizes.