Google

Causal Language Modeling (CLM), also known as conventional language modeling, can be understood as a specific instance of Masked Language Modeling (MLM). In CLM, the prediction of a token at a given position is constrained by masking all subsequent tokens in the right-hand context. This forces the model to rely exclusively on the preceding left-hand context, making it a unidirectional process. In contrast, the general MLM approach is bidirectional because it uses all unmasked tokens—from both the left and right contexts—to predict a masked token within a sequence.

Comparison of Masked vs. Causal Language Modeling

A language model is being developed specifically for a task that involves generating long, coherent passages of text, such as writing a story from an initial prompt. The model must generate the text sequentially, predicting each new word based only on the words that came before it. Which training approach is inherently structured for this type of task, and what is the key reason?

An AI research team is training a language model. During a specific training step, the model's objective is to predict the token at position `i` in a sequence. To do this, the model is only given access to the tokens from the beginning of the sequence up to position `i-1`. Information from any tokens at or after position `i` is deliberately withheld for this prediction. Based on this description of the training objective, which language modeling approach is being implemented? Justify your answer by explaining how the model's access to contextual information defines the approach.

Identifying Language Modeling Approach

Match each characteristic to the language modeling approach it describes. The two approaches are 'Causal Language Modeling' and 'General Masked Language Modeling'.

Learn Before

Related