Learn Before
Analyzing Pre-training Objectives
A research team is developing a model intended to be highly effective at summarizing long legal documents. They decide on a pre-training strategy where the model is given the first 90% of a document and is trained to produce the final 10% as its output. Analyze this pre-training strategy and determine which of the two primary problem types it represents. Justify your answer based on the model's objective.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Foundations of Large Language Models
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Sequence Encoding Models
Sequence Generation Models
Architectural Differences Between Sequence Encoding and Generation Models
General Formulation of a Sequence Model
A large language model is pre-trained on a vast text corpus. Its training objective is to take a sentence, randomly mask 15% of the words, and then predict only the original masked words by looking at all the surrounding unmasked words (both to the left and right). Which statement best analyzes the primary goal of this specific pre-training approach?
Analyzing Pre-training Objectives
Match each Natural Language Processing (NLP) task with the primary pre-training problem type it is designed to solve.