Learning Objectives of a Sentence Reconstruction Task
A language model is trained on a task where it must reconstruct a grammatically correct sentence from a version where all the words have been completely shuffled (e.g., turning is cat the sleeping into The cat is sleeping). Beyond simply learning vocabulary, what two key aspects of language structure does this specific training task force the model to learn?
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A language model is given a training task where it must reconstruct the original, grammatically correct sentence from a version where all the words and punctuation have been randomly shuffled. If the model receives the input sequence
ate mouse cheese the The ., which of the following is the correct output it should produce?Learning Objectives of a Sentence Reconstruction Task
Evaluating Reconstruction Difficulty