Learn Before
The Variable-Length Prediction Problem
A text sequence is modified by replacing a contiguous group of several words with a single, special placeholder token. When a language model is trained to restore the original text from this modified input, what specific information must the model learn to predict about the placeholder token, beyond just the meaning of the missing words? Explain why this is a necessary part of the restoration process.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
Analogy between Span Length Prediction and Fertility Modeling
A language model is tasked with filling in a masked portion of text. The input provided is: 'The solar system consists of the Sun and the
[MASK]that orbit it.' The original, unmasked text was: 'The solar system consists of the Sun and the celestial bodies that orbit it.' The model, however, generates the following output: 'The solar system consists of the Sun and the planets that orbit it.' Which statement provides the most accurate analysis of the model's primary error in this specific case?The Variable-Length Prediction Problem
Diagnosing a Model's Infilling Errors