Diagnosing Pre-training Deficiencies
A development team has pre-trained an encoder-decoder model using a denoising objective. The only method used to corrupt the input text was to replace a random 15% of the words with a special placeholder symbol. When fine-tuning this model for a paraphrasing task, they observe a significant weakness: the model struggles to recognize that two sentences with different word orders can have the same meaning (e.g., 'The team celebrated the victory' vs. 'The victory was celebrated by the team'). Based on this observation, what is a likely deficiency in the model's pre-training, and which specific input corruption strategy could be introduced to mitigate this issue? Justify your answer.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Token Masking as an Input Corruption Method
Token Deletion as an Input Corruption Method
Combining Multiple Corruption Methods in Pre-training
Selecting Appropriate Input Corruption Methods
Token Alteration as an Input Corruption Method
Token Reordering as an Input Corruption Method
Input Corruption Methods for Multi-Sentence Sequences
Input Corruption Methods for Multi-Sentence Sequences
Corruption Methods for Multi-Sentence Sequences
A research team is pre-training an encoder-decoder model using a denoising objective. Their primary goal is to create a model that excels at summarizing long documents, which requires a deep understanding of the text's overall semantic content and logical flow, rather than its exact word-for-word structure. Which of the following input corruption strategies would be most aligned with this specific goal?
You are training an encoder-decoder model with a denoising objective. Match each input corruption method with the primary linguistic capability it is designed to teach the model.
Diagnosing Pre-training Deficiencies