Google

For input sequences that contain multiple sentences, additional corruption techniques can be employed beyond standard token-level modifications. The BART model, for instance, incorporates two such methods specifically designed for multi-sentence text.

Input Corruption Methods for Multi-Sentence Sequences

A data scientist is training a model to generate concise summaries of multi-sentence news articles. The training process involves showing the model a 'damaged' version of an article and teaching it to reconstruct the original, complete article. The data scientist is considering two different methods for 'damaging' the input articles:

*   **Method 1:** Randomly reordering the sentences within each article, but keeping the words within each sentence in their original order.
*   **Method 2:** Keeping the sentences in their original order, but randomly replacing 15% of the individual words throughout the article with a special placeholder symbol.

Analyze the two methods. Which method is more likely to train a model that is better at the final goal of summarization, and why? Your explanation should compare how each method helps the model learn different aspects of language.

Selecting a Training Method for a Summarization Model

When training a model on a document with multiple sentences, what is the primary advantage of corrupting the input by randomly shuffling the order of entire sentences, as opposed to simply reordering individual tokens across the entire document?

A key pre-training strategy for models designed to understand long documents involves corrupting the input text. Consider two distinct corruption methods: (1) randomly shuffling the order of complete sentences within a document, and (2) replacing a span of text, which could include parts of sentences or multiple full sentences, with a single special token. Analyze how these two methods would likely differ in the types of linguistic understanding they encourage the model to develop.

Learn Before

Related