To enable PTMs to learn domain-specific knowledge, incorporating domain knowledge from external knowledge bases into PTM has been shown to be effective. 

On the one hand, external knowledge can be injected during pre-training. On the other hand, one can incorporate external knowledge into pre-trained models without retraining them from scratch.

Knowledge-Enriched PTMs

Learning multilingual text representations shared across languages plays an important role in many cross-lingual NLP tasks.
Examples:

- Multilingual BERT3 (mBERT): It is pre-trained by MLM with the shared vocabulary and weights on Wikipedia text from the top 104 languages. Each training sample is a monolingual document, and there are no cross-lingual objectives specifically designed nor any cross-lingual data. Even so, mBERT performs cross-lingual generalization surprisingly well.

- Cross-Lingual Language Model (XLM): XLM improves mBERT by incorporating a cross-lingual task, translation language modeling (TLM), which performs MLM on a concatenation of parallel bilingual sentence pairs.

Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results. For Chinese, which does not have explicit word boundaries, modeling larger granularity and multigranularity word representations have shown great success.
Some monolingual PTMs have been released for different languages, such as CamemBERT and FlauBERT for French, Fin-BERT for Finnish, BERTje and RobBERT for
Dutch, AraBERT for Arabic language.

Multilingual and Language-Specific PTMs

A great majority of these models are designed for a general visual and linguistic feature encoding. These models are pre-trained on some huge corpus of cross-modal data, such as videos with spoken words or images with captions, incorporating extended pre-training tasks to fully utilize the multi-modal feature. 


Typically, tasks like visual-based MLM, masked visual-feature modeling and visual-linguistic matching are widely used in multi-modal pre-training.

- Video-Text PTMs: To obtain sequences of visual and linguistic tokens used for pre-training, the videos are pre-processed by CNN-based encoders and off-the-shelf speech recognition techniques, respectively. And a single Transformer encoder is trained on the processed data to learn the vision-language representations for downstream tasks like video caption.

- Image-Text PTMs: several works introduce PTMs on image-text pairs, aiming to fit downstream tasks like visual question answering(VQA) and visual commonsense reasoning(VCR).

- Audio-Text PTMs: Several methods have explored the chance of PTMs on audio-text pairs, such as SpeechBERT. This work tries to build an end-to-end Speech Question Answering (SQA) model by encoding audio and text with a single Transformer encoder, which is pre-trained with MLM on speech and text corpus and fine-tuned on Question Answering.

Multi-Modal PTMs

Since pre-trained models (PTMs) usually consist of hundreds of millions of parameters, they are difficult to deploy in online services for real-world applications or on resource-restricted devices. Model compression is an approach used to reduce the model size and increase computational efficiency.

Compression of Pre-trained Models

There are several types of extensions of pre-trained models to better fit specific needs: Knowledge-Enriched PTMs, Multilingual and Language-Specific PTMs, Multi-Modal PTMs, Domain-Specific and Task-Specific PTMs, Model Compression, etc.

University of Michigan - Ann Arbor

Pre-training tasks are fundamental for developing universal language representations. After a model undergoes pre-training, various techniques are employed to adapt it for specific downstream applications. The pre-training tasks themselves are generally classified into three main types:

- **Supervised learning:** Training on data with explicit input-output pairs.
- **Unsupervised learning:** Discovering inherent knowledge from unlabeled data, such as through probabilistic language modeling.
- **Self-Supervised learning:** A hybrid approach where supervision signals are automatically generated from the data itself, as seen in masked language modeling (MLM).

Pre-training tasks

Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10), 1872-1897.

https://arxiv.org/abs/2003.08271. 

Pre-trained Models for Natural Language Processing: A Survey

It assumes some observed pairs of text that are more semantically similar than randomly sampled text.

There is a score function $s(x,y)$ for a text pair $(x,y)$ learned to minimize the objective function, where $(x, y^{+})$ are a similar pair and $y^{-}$ is presumably dissimilar to $x$.

Contrastive Learning (CTL)

Extensions of PTMs

A fundamental issue following the pre-training phase is applying the generalized pre-trained model, denoted as $$g_{\hat{\theta}}(\cdot)$$, to specific downstream tasks. To successfully adapt the model to these downstream tasks, it is necessary to slightly adjust its parameters, denoted as $$\hat{\theta}$$, using labeled data, or alternatively, to prompt the model with task descriptions.

Applying and Adapting Pre-trained Models to Downstream Tasks

Unsupervised pre-training, a key focus in the early resurgence of deep learning, involves optimizing a neural network's parameters using a task-agnostic criterion. Instead of relying on task-specific labels, this method uses objectives like minimizing the input's reconstruction cross-entropy. It is typically used as a preparatory phase before a model undergoes supervised training.

Unsupervised Pre-training

Supervised pre-training is an approach where a neural network, such as a sequence model designed to encode inputs into representations, is initially trained on a supervised task. This is achieved by combining the core model with a classification layer to form a complete system. This system is then trained on a labeled dataset for a specific pre-training objective, like sentiment classification, before it is adapted for other downstream tasks.

Supervised Pre-training

Self-supervised learning is a pre-training method where a neural network is trained using supervision signals it generates from the data itself, rather than from human-provided labels. This is accomplished by constructing training tasks directly from unlabeled data, for example, by having the system create its own pseudo-labels. This approach facilitates large-scale training for deep neural networks, which has proven highly successful for developing models proficient in various understanding, writing, and reasoning tasks.

Self-Supervised Learning

Pre-training paradigms differ primarily in their data requirements, underlying assumptions, and adaptation methods. Unsupervised pre-training utilizes large-scale unlabeled data to create a good preliminary starting point, though it still requires considerable effort to further train the model with labeled data. Supervised pre-training relies on labeled datasets from the outset, operating on the assumption that different tasks are related, which allows a model trained on one task to be transferred to another via tuning. In contrast, self-supervised pre-training leverages large-scale unlabeled data to train a model that can be efficiently adapted to new tasks through methods like fine-tuning or prompting.

Comparison of Pre-training Paradigms

Categorizing pre-training tasks based purely on model architecture is suboptimal because a single training objective can be applied across varying architectures (e.g., masked language modeling works for both encoder-only and encoder-decoder setups). Therefore, a better approach is to classify these tasks based on their specific training objectives.

Rationale for Categorizing Pre-training Tasks by Objective

Denoising autoencoding is a pre-training approach, often used for encoder-decoder models, that trains a model to reconstruct an original, clean sequence from a corrupted input. The core task is to learn robust representations by removing the noise that was artificially introduced into the data.

Denoising Autoencoding

Despite their differences, various pre-training tasks can be evaluated and compared against one another by utilizing a consistent framework and a standardized experimental setup.

Comparability of Pre-training Tasks

The effectiveness of pre-trained models stems from the general nature of their training tasks. By learning from broad objectives rather than task-specific ones, these models build versatile representations. This generalist foundation enables them to achieve strong performance across a wide variety of NLP problems, often to the point of outperforming systems that were previously developed with specialized, supervised training for individual tasks.

Generality of Pre-training Tasks and Performance

Once a model has undergone pre-training, various techniques are necessary to adapt it for specific downstream applications. The selection of an appropriate adaptation method is a key step in leveraging pre-trained models effectively.

Applying Pre-trained Models to Downstream Tasks

Given the following training scenario, identify which category of pre-training task is being used (Supervised, Unsupervised, or Self-Supervised) and justify your answer by explaining how the training data and labels are handled.

Identifying a Pre-training Strategy

The landscape of pre-training includes a vast and diverse array of tasks, making an exhaustive list impractical. For deeper and more comprehensive discussions on the variety of available pre-training tasks, specialized survey papers on the topic are a valuable resource.

Breadth of Pre-training Tasks

A research team is developing a new language model and is considering different pre-training approaches. Match each pre-training scenario below with the correct category of learning it represents.

A language model is being trained on a large corpus of text from the internet. The training process involves randomly hiding 15% of the words in each sentence and then tasking the model with predicting the original identity of these hidden words based on the surrounding context. Which category of pre-training task does this scenario best exemplify, and why?

A machine learning engineer is tasked with developing a language model but has access to different types of datasets. Compare and contrast the three main categories of pre-training tasks (supervised, unsupervised, and self-supervised) in terms of their data requirements and the nature of their supervision signals. For each category, provide a hypothetical example of a task and dataset that would be appropriate.

Comparing Pre-training Task Categories

Pre-training tasks can be compared based on their input-output transformations and their applicability to specific model architectures. Language modeling variants (such as Causal and Prefix LM) focus on sequential text generation and are typically applied to decoder-only and encoder-decoder models. Masked language modeling approaches (e.g., MASS-style, BERT-style) rely on reconstructing masked tokens and are compatible with both encoder-only and encoder-decoder architectures. Permuted language modeling and discriminative training methods (like Next Sentence Prediction, Sentence Comparison, and Token Classification) are specifically tailored for encoder-only models. Finally, denoising autoencoding encompasses tasks such as token reordering, token deletion, span masking, sentinel masking, sentence reordering, and document rotation, which train encoder-decoder models to reconstruct original text from corrupted inputs.

Learn Before

Related

Learn After