Token Deletion as an Input Corruption Method
Token deletion is an input corruption technique where certain tokens are randomly selected from an input sequence and then completely removed. This method is distinct from token masking, as the selected tokens are not replaced with a special [MASK] symbol but are instead deleted from the sequence, resulting in a shorter input.
0
1
References
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Reference of Foundations of Large Language Models Course
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Token Masking as an Input Corruption Method
Token Deletion as an Input Corruption Method
Combining Multiple Corruption Methods in Pre-training
Selecting Appropriate Input Corruption Methods
Token Alteration as an Input Corruption Method
Token Reordering as an Input Corruption Method
Input Corruption Methods for Multi-Sentence Sequences
Input Corruption Methods for Multi-Sentence Sequences
Corruption Methods for Multi-Sentence Sequences
A research team is pre-training an encoder-decoder model using a denoising objective. Their primary goal is to create a model that excels at summarizing long documents, which requires a deep understanding of the text's overall semantic content and logical flow, rather than its exact word-for-word structure. Which of the following input corruption strategies would be most aligned with this specific goal?
You are training an encoder-decoder model with a denoising objective. Match each input corruption method with the primary linguistic capability it is designed to teach the model.
Diagnosing Pre-training Deficiencies
Learn After
Example Comparison of Token Masking and Token Deletion
A language model is being trained to reconstruct an original text sequence from a corrupted version. During one training step, the original input is 'The quick brown fox jumps over the lazy dog.' and the corrupted input given to the model is 'The quick fox over the lazy dog.'. Based on this example, which specific input corruption technique was applied?
Analysis of Input Corruption Impact
When applying the token deletion method to corrupt an input sequence for model training, the length of the resulting sequence is identical to the original sequence.
Example of Token Deletion in Denoising Autoencoding