Learn Before
Divergent Pre-training Paradigms in NLP and Computer Vision
A key distinction in the historical development of pre-training lies in the different approaches adopted by the fields of computer vision and Natural Language Processing. The standard paradigm in computer vision involved supervised pre-training, where models were trained on large, manually labeled datasets such as ImageNet. In contrast, the breakthrough in modern NLP was driven by large-scale, self-supervised learning, which leverages vast quantities of unlabeled text.
0
1
Tags
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
A key development leading to modern large-scale language models was the adoption of a pre-training paradigm. How did the influential pre-training approach that became standard in computer vision fundamentally differ from the self-supervised approach that now dominates natural language processing?
Arrange the following key developments in the history of pre-training into the correct chronological and influential sequence, from the earliest concept to the modern paradigm.
Converging Histories of Pre-training
Divergent Pre-training Paradigms in NLP and Computer Vision
Self-Supervised Pre-training of Language Models