Google

For text inputs that span multiple sentences, standard token-level corruption can be supplemented with sentence-level techniques. The BART model, for example, utilizes two such methods to corrupt multi-sentence documents.

Input Corruption Methods for Multi-Sentence Sequences

Sentence reordering is a data corruption technique applied to multi-sentence inputs where the original order of sentences within a document is randomly shuffled. The model is then trained on this permuted sequence with the objective of learning to restore the sentences to their correct, logical order.

Sentence Reordering as an Input Corruption Method

Document rotation is an input corruption method where the primary objective is for a model to identify the original start of a sequence. The process begins by randomly selecting a token from the input text. The entire sequence is then rotated so that this selected token is positioned at the beginning, creating a corrupted version. The model is then trained on this rotated sequence to predict which token was originally the first one.

Document Rotation as an Input Corruption Method

A research team is training a model on multi-paragraph documents. Their primary goal is to ensure the model learns the logical flow and coherence between sentences, not just the relationships between words within a single sentence. Which of the following input corruption strategies is specifically designed to target this higher-level, inter-sentence understanding?

A language model is being trained on multi-sentence documents, such as news articles, using a denoising objective. The training data is already being corrupted by masking random individual words. Explain why it would be beneficial to also introduce corruption methods that operate at the sentence level.

Rationale for Sentence-Level Corruption

A language model is being trained using a denoising objective, where it learns to reconstruct original text from a corrupted version. Match each type of input corruption with the primary linguistic feature it forces the model to learn.

Learn Before

Related