Learn Before
BART Model's Corruption Methods for Multi-Sentence Sequences
The BART (Bidirectional and Auto-Regressive Transformers) model employs two specific corruption strategies that are designed for sequences containing multiple sentences.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
BART Model's Corruption Methods for Multi-Sentence Sequences
When pre-training a model on a document, a common strategy is to intentionally alter the input text and task the model with restoring the original. Which of the following alteration techniques is uniquely dependent on the input text containing more than one sentence?
When preparing text data to train a language model, various 'corruption' techniques are used to alter the original input, which the model then learns to restore. Some of these techniques operate on the word or token level, while others operate on the sentence level. Match each corruption technique described below with the structural requirement of the input text.
Analyzing Text Corruption Strategies
Learn After
Sentence Reordering as an Input Corruption Method
A pre-training process is applied to the following two-sentence input: 'The team celebrated their victory. They had trained hard for months.' The input is transformed into: 'The team [MASK] hard for months.' The model is then tasked with reconstructing the original, complete text from this transformed input. Which specific data corruption technique, designed for handling sequences of text, does this process exemplify?
Choosing a Pre-training Strategy
A model is pre-trained using corruption techniques that operate on the structure of multi-sentence documents. Match each technique with its correct description.