Selecting a Training Method for a Summarization Model
A data scientist is training a model to generate concise summaries of multi-sentence news articles. The training process involves showing the model a 'damaged' version of an article and teaching it to reconstruct the original, complete article. The data scientist is considering two different methods for 'damaging' the input articles:
- Method 1: Randomly reordering the sentences within each article, but keeping the words within each sentence in their original order.
- Method 2: Keeping the sentences in their original order, but randomly replacing 15% of the individual words throughout the article with a special placeholder symbol.
Analyze the two methods. Which method is more likely to train a model that is better at the final goal of summarization, and why? Your explanation should compare how each method helps the model learn different aspects of language.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Selecting a Training Method for a Summarization Model
When training a model on a document with multiple sentences, what is the primary advantage of corrupting the input by randomly shuffling the order of entire sentences, as opposed to simply reordering individual tokens across the entire document?
Comparing Multi-Sentence Corruption Techniques