Learn Before
Converging Histories of Pre-training
The pre-training paradigm that powers modern large language models was influenced by separate, parallel developments. Analyze the two primary historical streams of pre-training—one that became standard in computer vision and one rooted in early unsupervised learning for tasks like word embeddings—and explain how distinct elements from both contributed to the large-scale, self-supervised approach dominant in natural language processing today.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Foundations of Large Language Models
Analysis in Bloom's Taxonomy
Cognitive Psychology
Psychology
Social Science
Empirical Science
Science
Related
A key development leading to modern large-scale language models was the adoption of a pre-training paradigm. How did the influential pre-training approach that became standard in computer vision fundamentally differ from the self-supervised approach that now dominates natural language processing?
Arrange the following key developments in the history of pre-training into the correct chronological and influential sequence, from the earliest concept to the modern paradigm.
Converging Histories of Pre-training
Divergent Pre-training Paradigms in NLP and Computer Vision
Self-Supervised Pre-training of Language Models