Analysis of Input Corruption Techniques
Compare and contrast two common methods for corrupting text inputs during the pre-training of a denoising autoencoder: 'Token Masking' (where individual tokens are replaced with a special symbol) and 'Text Infilling' (where a span of text of variable length is replaced with a single special symbol). In your analysis, explain how each method forces the model to learn different aspects of language structure and meaning.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Diagnosing a Denoising Pre-training Strategy
A research team is pre-training a text-based model with the goal of making it highly robust and flexible for a wide range of downstream applications, including generating coherent paragraphs and correcting grammatical errors. The model is trained to reconstruct original text from a corrupted version. Which of the following corruption strategies applied during pre-training would be most effective for achieving this goal?
Analysis of Input Corruption Techniques