Learn Before
Example of Document Rotation in Denoising Autoencoding
Document rotation is a denoising autoencoding technique where an encoder-decoder model learns to reconstruct a text sequence that has been cyclically shifted. For example, if a document is corrupted into [C] chasing the ball . The ball rolls away swiftly . The kitten is, the model is tasked with identifying the true starting point and generating the sequence in its proper order: The kitten is chasing the ball . The ball rolls away swiftly ..
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Example of Document Rotation
A self-supervised learning task involves modifying an input sequence by selecting a token and rearranging the sequence so that the selected token becomes the new starting point. The part of the sequence that originally came before the selected token is moved to the end. Given the original sequence 'Hard work leads to success .', if the token 'leads' is chosen as the new starting point, what is the resulting modified sequence?
Reconstructing Original Sequence from Rotated Input
A language model is being trained using a technique where an input document is 'rotated'. For example, an original document is transformed into the following sequence: 'leads to success . Success brings happiness . Hard work'. What is the primary objective for the model when presented with this transformed input?
Example of Document Rotation in Denoising Autoencoding