Learn Before
Analogy between Span Length Prediction and Fertility Modeling
The problem of determining how many tokens to generate from a single masked span is conceptually similar to fertility modeling in the field of machine translation.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Computing Sciences
Foundations of Large Language Models Course
Related
Analogy between Span Length Prediction and Fertility Modeling
A language model is tasked with filling in a masked portion of text. The input provided is: 'The solar system consists of the Sun and the
[MASK]that orbit it.' The original, unmasked text was: 'The solar system consists of the Sun and the celestial bodies that orbit it.' The model, however, generates the following output: 'The solar system consists of the Sun and the planets that orbit it.' Which statement provides the most accurate analysis of the model's primary error in this specific case?The Variable-Length Prediction Problem
Diagnosing a Model's Infilling Errors
Learn After
Conceptual Parallel in Generative Modeling
In generative language models, the challenge of predicting the length of a masked span is often compared to the concept of 'fertility' from statistical machine translation. Which of the following statements most accurately analyzes the relationship between these two concepts?
Critique of the Fertility Modeling Analogy