Consider the task of predicting the token 'fox' in the sequence 'The quick brown fox jumps'. To make a bidirectional model's prediction for 'fox' equivalent to that of a unidirectional (left-to-right) model, which set of tokens must be masked (i.e., hidden) from the bidirectional model's view?

Adapting a Bidirectional Model for a Unidirectional Task

A language model trained exclusively for next-token prediction (i.e., predicting a word based only on the words that precede it) can be framed as a specific implementation of a masked language model where, for every prediction, all subsequent tokens in the sequence are systematically masked.

Adapting a Bidirectional Model for Generative Tasks