Adapting a Bidirectional Model for a Unidirectional Task
A researcher has a language model that was trained to use both preceding and succeeding text to fill in missing words. They now want to use this model for a text generation task, where the goal is to predict the next word in a sentence, one word at a time. Describe the specific masking strategy the researcher must apply to the input sequence at each step of the generation process to force the model to only use the context it has already generated.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Consider the task of predicting the token 'fox' in the sequence 'The quick brown fox jumps'. To make a bidirectional model's prediction for 'fox' equivalent to that of a unidirectional (left-to-right) model, which set of tokens must be masked (i.e., hidden) from the bidirectional model's view?
Adapting a Bidirectional Model for a Unidirectional Task
A language model trained exclusively for next-token prediction (i.e., predicting a word based only on the words that precede it) can be framed as a specific implementation of a masked language model where, for every prediction, all subsequent tokens in the sequence are systematically masked.
Adapting a Bidirectional Model for Generative Tasks