Google

Pre-training tasks are fundamental for developing universal language representations. After a model undergoes pre-training, various techniques are employed to adapt it for specific downstream applications. The pre-training tasks themselves are generally classified into three main types:

- **Supervised learning:** Training on data with explicit input-output pairs.
- **Unsupervised learning:** Discovering inherent knowledge from unlabeled data, such as through probabilistic language modeling.
- **Self-Supervised learning:** A hybrid approach where supervision signals are automatically generated from the data itself, as seen in masked language modeling (MLM).

Pre-training tasks

It assumes some observed pairs of text that are more semantically similar than randomly sampled text.

There is a score function $s(x,y)$ for a text pair $(x,y)$ learned to minimize the objective function, where $(x, y^{+})$ are a similar pair and $y^{-}$ is presumably dissimilar to $x$.

Contrastive Learning (CTL)

There are several types of extensions of pre-trained models to better fit specific needs: Knowledge-Enriched PTMs, Multilingual and Language-Specific PTMs, Multi-Modal PTMs, Domain-Specific and Task-Specific PTMs, Model Compression, etc.

Extensions of PTMs

A fundamental issue following the pre-training phase is applying the generalized pre-trained model, denoted as $$g_{\hat{\theta}}(\cdot)$$, to specific downstream tasks. To successfully adapt the model to these downstream tasks, it is necessary to slightly adjust its parameters, denoted as $$\hat{\theta}$$, using labeled data, or alternatively, to prompt the model with task descriptions.

Applying and Adapting Pre-trained Models to Downstream Tasks

Unsupervised pre-training, a key focus in the early resurgence of deep learning, involves optimizing a neural network's parameters using a task-agnostic criterion. Instead of relying on task-specific labels, this method uses objectives like minimizing the input's reconstruction cross-entropy. It is typically used as a preparatory phase before a model undergoes supervised training.

Unsupervised Pre-training

Supervised pre-training is an approach where a neural network, such as a sequence model designed to encode inputs into representations, is initially trained on a supervised task. This is achieved by combining the core model with a classification layer to form a complete system. This system is then trained on a labeled dataset for a specific pre-training objective, like sentiment classification, before it is adapted for other downstream tasks.

Supervised Pre-training

Self-supervised learning is a pre-training method where a neural network is trained using supervision signals it generates from the data itself, rather than from human-provided labels. This is accomplished by constructing training tasks directly from unlabeled data, for example, by having the system create its own pseudo-labels. This approach facilitates large-scale training for deep neural networks, which has proven highly successful for developing models proficient in various understanding, writing, and reasoning tasks.

Self-Supervised Learning

Pre-training paradigms differ primarily in their data requirements, underlying assumptions, and adaptation methods. Unsupervised pre-training utilizes large-scale unlabeled data to create a good preliminary starting point, though it still requires considerable effort to further train the model with labeled data. Supervised pre-training relies on labeled datasets from the outset, operating on the assumption that different tasks are related, which allows a model trained on one task to be transferred to another via tuning. In contrast, self-supervised pre-training leverages large-scale unlabeled data to train a model that can be efficiently adapted to new tasks through methods like fine-tuning or prompting.

Comparison of Pre-training Paradigms

Categorizing pre-training tasks based purely on model architecture is suboptimal because a single training objective can be applied across varying architectures (e.g., masked language modeling works for both encoder-only and encoder-decoder setups). Therefore, a better approach is to classify these tasks based on their specific training objectives.

Rationale for Categorizing Pre-training Tasks by Objective

Denoising autoencoding is a pre-training approach, often used for encoder-decoder models, that trains a model to reconstruct an original, clean sequence from a corrupted input. The core task is to learn robust representations by removing the noise that was artificially introduced into the data.

Denoising Autoencoding

Despite their differences, various pre-training tasks can be evaluated and compared against one another by utilizing a consistent framework and a standardized experimental setup.

Comparability of Pre-training Tasks

The effectiveness of pre-trained models stems from the general nature of their training tasks. By learning from broad objectives rather than task-specific ones, these models build versatile representations. This generalist foundation enables them to achieve strong performance across a wide variety of NLP problems, often to the point of outperforming systems that were previously developed with specialized, supervised training for individual tasks.

Generality of Pre-training Tasks and Performance

Once a model has undergone pre-training, various techniques are necessary to adapt it for specific downstream applications. The selection of an appropriate adaptation method is a key step in leveraging pre-trained models effectively.

Applying Pre-trained Models to Downstream Tasks

Given the following training scenario, identify which category of pre-training task is being used (Supervised, Unsupervised, or Self-Supervised) and justify your answer by explaining how the training data and labels are handled.

Identifying a Pre-training Strategy

The landscape of pre-training includes a vast and diverse array of tasks, making an exhaustive list impractical. For deeper and more comprehensive discussions on the variety of available pre-training tasks, specialized survey papers on the topic are a valuable resource.

Breadth of Pre-training Tasks

A research team is developing a new language model and is considering different pre-training approaches. Match each pre-training scenario below with the correct category of learning it represents.

A language model is being trained on a large corpus of text from the internet. The training process involves randomly hiding 15% of the words in each sentence and then tasking the model with predicting the original identity of these hidden words based on the surrounding context. Which category of pre-training task does this scenario best exemplify, and why?

A machine learning engineer is tasked with developing a language model but has access to different types of datasets. Compare and contrast the three main categories of pre-training tasks (supervised, unsupervised, and self-supervised) in terms of their data requirements and the nature of their supervision signals. For each category, provide a hypothetical example of a task and dataset that would be appropriate.

Comparing Pre-training Task Categories

Pre-training tasks can be compared based on their input-output transformations and their applicability to specific model architectures. Language modeling variants (such as Causal and Prefix LM) focus on sequential text generation and are typically applied to decoder-only and encoder-decoder models. Masked language modeling approaches (e.g., MASS-style, BERT-style) rely on reconstructing masked tokens and are compatible with both encoder-only and encoder-decoder architectures. Permuted language modeling and discriminative training methods (like Next Sentence Prediction, Sentence Comparison, and Token Classification) are specifically tailored for encoder-only models. Finally, denoising autoencoding encompasses tasks such as token reordering, token deletion, span masking, sentinel masking, sentence reordering, and document rotation, which train encoder-decoder models to reconstruct original text from corrupted inputs.

Learn Before

Related