Corruption Methods for Multi-Sentence Sequences
When an input sequence is composed of multiple sentences, it becomes possible to apply additional types of corruption beyond token-level alterations. These methods operate on a larger scale, leveraging the multi-sentence structure of the text.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Token Masking as an Input Corruption Method
Token Deletion as an Input Corruption Method
Combining Multiple Corruption Methods in Pre-training
Selecting Appropriate Input Corruption Methods
Token Alteration as an Input Corruption Method
Token Reordering as an Input Corruption Method
Input Corruption Methods for Multi-Sentence Sequences
Input Corruption Methods for Multi-Sentence Sequences
Corruption Methods for Multi-Sentence Sequences
A research team is pre-training an encoder-decoder model using a denoising objective. Their primary goal is to create a model that excels at summarizing long documents, which requires a deep understanding of the text's overall semantic content and logical flow, rather than its exact word-for-word structure. Which of the following input corruption strategies would be most aligned with this specific goal?
You are training an encoder-decoder model with a denoising objective. Match each input corruption method with the primary linguistic capability it is designed to teach the model.
Diagnosing Pre-training Deficiencies
Learn After
BART Model's Corruption Methods for Multi-Sentence Sequences
When pre-training a model on a document, a common strategy is to intentionally alter the input text and task the model with restoring the original. Which of the following alteration techniques is uniquely dependent on the input text containing more than one sentence?
When preparing text data to train a language model, various 'corruption' techniques are used to alter the original input, which the model then learns to restore. Some of these techniques operate on the word or token level, while others operate on the sentence level. Match each corruption technique described below with the structural requirement of the input text.
Analyzing Text Corruption Strategies