Adapting a Bidirectional Model for Generative Tasks
Describe the specific masking strategy a research team must apply to their input sequences during fine-tuning to force a pre-trained bidirectional model to behave like a unidirectional, causal language model for a text generation task. For a given token position i that needs to be predicted, which other tokens in the sequence should be masked?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Application in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Consider the task of predicting the token 'fox' in the sequence 'The quick brown fox jumps'. To make a bidirectional model's prediction for 'fox' equivalent to that of a unidirectional (left-to-right) model, which set of tokens must be masked (i.e., hidden) from the bidirectional model's view?
Adapting a Bidirectional Model for a Unidirectional Task
A language model trained exclusively for next-token prediction (i.e., predicting a word based only on the words that precede it) can be framed as a specific implementation of a masked language model where, for every prediction, all subsequent tokens in the sequence are systematically masked.
Adapting a Bidirectional Model for Generative Tasks