Learn Before
Assumption of Supervised Pre-training
The underlying assumption in supervised pre-training is that different supervised learning tasks share a relationship. Because of this connection, a neural network can be initially trained on one specific task and then effectively transferred to a different task, requiring only some additional tuning or training effort to adapt to the new objective.
0
1
Tags
Foundations of Large Language Models
Ch.1 Pre-training - Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Process of Adapting a Supervised Pre-trained Model
Advantages of Supervised Pre-training
Disadvantages of Supervised Pre-training
Example of a Supervised Pre-training Task
A startup is building a system to automatically categorize legal contracts into specific sub-types (e.g., 'lease agreement', 'employment contract', 'non-disclosure agreement'). They have a very small, private dataset of 500 labeled contracts. Their proposed strategy is to first train a large neural network on a massive, publicly available dataset of millions of labeled news articles, classifying them by topic (e.g., 'sports', 'politics', 'technology'). After this initial training, they plan to a
A machine learning engineer wants to use a supervised pre-training approach to build a model that can detect toxic comments online. Arrange the following steps in the correct chronological order to reflect this process.
Evaluating a Pre-training Strategy for Scientific Text
Assumption of Supervised Pre-training