Example Comparison of Token Masking and Token Deletion

A language model is being trained to reconstruct an original text sequence from a corrupted version. During one training step, the original input is 'The quick brown fox jumps over the lazy dog.' and the corrupted input given to the model is 'The quick fox over the lazy dog.'. Based on this example, which specific input corruption technique was applied?

Analysis of Input Corruption Impact

When applying the token deletion method to corrupt an input sequence for model training, the length of the resulting sequence is identical to the original sequence.

Example of Token Deletion in Denoising Autoencoding