To enable PTMs to learn domain-specific knowledge, incorporating domain knowledge from external knowledge bases into PTM has been shown to be effective. 

On the one hand, external knowledge can be injected during pre-training. On the other hand, one can incorporate external knowledge into pre-trained models without retraining them from scratch.

University of Michigan - Ann Arbor

There are several types of extensions of pre-trained models to better fit specific needs: Knowledge-Enriched PTMs, Multilingual and Language-Specific PTMs, Multi-Modal PTMs, Domain-Specific and Task-Specific PTMs, Model Compression, etc.

Extensions of PTMs

Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020). Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10), 1872-1897.

https://arxiv.org/abs/2003.08271. 

Pre-trained Models for Natural Language Processing: A Survey

Knowledge-Enriched PTMs

Learning multilingual text representations shared across languages plays an important role in many cross-lingual NLP tasks.
Examples:

- Multilingual BERT3 (mBERT): It is pre-trained by MLM with the shared vocabulary and weights on Wikipedia text from the top 104 languages. Each training sample is a monolingual document, and there are no cross-lingual objectives specifically designed nor any cross-lingual data. Even so, mBERT performs cross-lingual generalization surprisingly well.

- Cross-Lingual Language Model (XLM): XLM improves mBERT by incorporating a cross-lingual task, translation language modeling (TLM), which performs MLM on a concatenation of parallel bilingual sentence pairs.

Although multilingual PTMs perform well on many languages, recent work showed that PTMs trained on a single language significantly outperform the multilingual results. For Chinese, which does not have explicit word boundaries, modeling larger granularity and multigranularity word representations have shown great success.
Some monolingual PTMs have been released for different languages, such as CamemBERT and FlauBERT for French, Fin-BERT for Finnish, BERTje and RobBERT for
Dutch, AraBERT for Arabic language.

Multilingual and Language-Specific PTMs

A great majority of these models are designed for a general visual and linguistic feature encoding. These models are pre-trained on some huge corpus of cross-modal data, such as videos with spoken words or images with captions, incorporating extended pre-training tasks to fully utilize the multi-modal feature. 


Typically, tasks like visual-based MLM, masked visual-feature modeling and visual-linguistic matching are widely used in multi-modal pre-training.

- Video-Text PTMs: To obtain sequences of visual and linguistic tokens used for pre-training, the videos are pre-processed by CNN-based encoders and off-the-shelf speech recognition techniques, respectively. And a single Transformer encoder is trained on the processed data to learn the vision-language representations for downstream tasks like video caption.

- Image-Text PTMs: several works introduce PTMs on image-text pairs, aiming to fit downstream tasks like visual question answering(VQA) and visual commonsense reasoning(VCR).

- Audio-Text PTMs: Several methods have explored the chance of PTMs on audio-text pairs, such as SpeechBERT. This work tries to build an end-to-end Speech Question Answering (SQA) model by encoding audio and text with a single Transformer encoder, which is pre-trained with MLM on speech and text corpus and fine-tuned on Question Answering.

Multi-Modal PTMs

Since pre-trained models (PTMs) usually consist of hundreds of millions of parameters, they are difficult to deploy in online services for real-world applications or on resource-restricted devices. Model compression is an approach used to reduce the model size and increase computational efficiency.

Learn Before

Related