Learn Before
Example of Token Reordering in Denoising Autoencoding
In denoising autoencoding, token reordering trains an encoder-decoder model to reconstruct a coherent sequence from an input where the words have been shuffled. For example, if a model is presented with the corrupted sequence [C] . kitten the chasing The is ball, it must learn to generate the original text in its correct order: The kitten is chasing the ball ., typically predicting each token autoregressively.
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
An engineer is training a model whose task is to reconstruct an original sentence from a modified version of it. The engineer's primary goal is to force the model to learn the semantic meaning of the sentence, independent of the specific ordering of its words. Which of the following modification techniques, when applied to the input sentence, would be most effective for achieving this specific training objective?
Comparing Input Alteration Techniques
Evaluating a Training Strategy for a Summarization Model
Example of Token Reordering in Denoising Autoencoding