A pre-training process is applied to the following two-sentence input: 'The team celebrated their victory. They had trained hard for months.' The input is transformed into: 'The team [MASK] hard for months.' The model is then tasked with reconstructing the original, complete text from this transformed input. Which specific data corruption technique, designed for handling sequences of text, does this process exemplify?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Sentence Reordering as an Input Corruption Method
A pre-training process is applied to the following two-sentence input: 'The team celebrated their victory. They had trained hard for months.' The input is transformed into: 'The team [MASK] hard for months.' The model is then tasked with reconstructing the original, complete text from this transformed input. Which specific data corruption technique, designed for handling sequences of text, does this process exemplify?
Choosing a Pre-training Strategy
A model is pre-trained using corruption techniques that operate on the structure of multi-sentence documents. Match each technique with its correct description.