Input Corruption Methods for Multi-Sentence Sequences
For text inputs that span multiple sentences, standard token-level corruption can be supplemented with sentence-level techniques. The BART model, for example, utilizes two such methods to corrupt multi-sentence documents.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Token Masking as an Input Corruption Method
Token Deletion as an Input Corruption Method
Combining Multiple Corruption Methods in Pre-training
Selecting Appropriate Input Corruption Methods
Token Alteration as an Input Corruption Method
Token Reordering as an Input Corruption Method
Input Corruption Methods for Multi-Sentence Sequences
Input Corruption Methods for Multi-Sentence Sequences
Corruption Methods for Multi-Sentence Sequences
A research team is pre-training an encoder-decoder model using a denoising objective. Their primary goal is to create a model that excels at summarizing long documents, which requires a deep understanding of the text's overall semantic content and logical flow, rather than its exact word-for-word structure. Which of the following input corruption strategies would be most aligned with this specific goal?
You are training an encoder-decoder model with a denoising objective. Match each input corruption method with the primary linguistic capability it is designed to teach the model.
Diagnosing Pre-training Deficiencies
Learn After
Sentence Reordering as an Input Corruption Method
Document Rotation as an Input Corruption Method
A research team is training a model on multi-paragraph documents. Their primary goal is to ensure the model learns the logical flow and coherence between sentences, not just the relationships between words within a single sentence. Which of the following input corruption strategies is specifically designed to target this higher-level, inter-sentence understanding?
Rationale for Sentence-Level Corruption
A language model is being trained using a denoising objective, where it learns to reconstruct original text from a corrupted version. Match each type of input corruption with the primary linguistic feature it forces the model to learn.