Unintended Learning in Sentence Relationship Models
A language model is trained on a task where it must determine if Sentence B is the actual sentence that follows Sentence A. For negative examples (where B is not the next sentence), the training data is constructed by always pairing Sentence A with a random sentence from a completely different document. Explain a potential superficial shortcut the model might learn from this setup, and why this shortcut fails to capture a true understanding of sentence coherence.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is being trained on a task where it must determine if Sentence B is the actual sentence that follows Sentence A in a document. Which of the following training pairs is most likely to encourage the model to learn a simple, superficial shortcut for this task, rather than developing a deeper understanding of semantic coherence?
Simplicity of NSP Task as a Cause for Reliance on Superficial Cues
Diagnosing a Language Model's Flawed Coherence Judgment
Unintended Learning in Sentence Relationship Models