After a pre-trained language model (PLM) produces a sequence of vectors for contextual representation, one or more task-specific layers are added to generate the final output. The architecture of these layers depends on the target task's required linguistic structure. For example, recurrent neural networks (RNNs) capture word order, convolutional neural networks (CNNs) recognize key phrase patterns, attention mechanisms identify correlated words, Siamese neural networks handle text matching, and graph neural networks (GNNs) utilize graph structures like parse trees.

Saint Olaf College

Google

Depends greatly on the nature of the target task and domain, the availability of in-domain labels, the latency and capacity constraints of the applications, and so on. Has 5 steps.

How to choose the best neural network model for my task.

The use of PLMs leads to significant improvements across all popular text classification tasks, and autoencoding PLMs (e.g., BERT or RoBERTa) often work better than autoregressive PLMs (e.g., OpenAI GPT). Hugging Face maintains a rich repository of PLMs developed for various tasks and settings.

 PLM Selection

Most PLMs are trained on general-domain text corpora, If the target domain is dramatically different from a general domain, we might consider adapting the PLM using in-domain data by continual pre-training the selected general-domain PLM. For domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch might also be a good choice.

Domain adaptation

Task-Specific Layers for Pre-Trained Language Models

Pre-trained language models (PLMs) are computationally expensive to deploy and serve. They often require compression, such as through knowledge distillation, to satisfy the latency and capacity constraints of real-world applications.

Learn Before

Related