Auto-regressive model is often formatted as:
$y(t) = \sum_{i=1}^n a_i y(t-i) + e(t)$
It uses linear combination of previous time-related variables to predict the variable in the time t.

Auto-regressive Model in NLP

It maps the input into a hidden state and try to reconstruct the input to get a better hidden state for representation learning.

Autoencoding Model

It means mapping one sequence to another sequence.

Seq2Seq Model

In the pre-training phase, an encoder model is trained using a self-supervision objective like Masked Language Modeling. The process begins by converting a corrupted input sequence, where some tokens are masked, into a sequence of embeddings. This embedding sequence is then fed into the encoder, which generates contextual vector representations for all input tokens. Finally, these representations are passed to an output layer, such as a Softmax model, which is trained to reconstruct the original masked tokens.

Self-Supervised Pre-training of Encoders via Masked Language Modeling

1. autoregressive-models
2. autoencoding-models
3. seq-to-seq-models
4. multimodal-models
5. retrieval-based-models
6. masked Language models

University of Michigan - Ann Arbor

University of Illinois at Urbana-Champaign

The pre-training and fine-tuning paradigm is a method motivated by the goal of creating adaptable, general-purpose systems for universal language understanding and generation. It involves separating the common components of neural network architectures, such as Transformers, and training them on vast amounts of unlabeled data using self-supervision. The resulting systems, known as foundation models, can be easily adapted for specific downstream applications via fine-tuning or prompting. This paradigm shift has enormously transformed natural language processing, meaning that in many cases, large-scale supervised learning for specific tasks is no longer required.

The Pre-training and Fine-tuning Paradigm

https://huggingface.co/docs/transformers/model_summary

Huggingface Model Summary

Ding, Ning et al. “Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.” ArXiv abs/2203.06904 (2022): n. pag. 
From  https://arxiv.org/pdf/2203.06904.pdf

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Types of Pretrained Language Model

Pre-training tasks are fundamental for developing universal language representations. After a model undergoes pre-training, various techniques are employed to adapt it for specific downstream applications. The pre-training tasks themselves are generally classified into three main types:

- **Supervised learning:** Training on data with explicit input-output pairs.
- **Unsupervised learning:** Discovering inherent knowledge from unlabeled data, such as through probabilistic language modeling.
- **Self-Supervised learning:** A hybrid approach where supervision signals are automatically generated from the data itself, as seen in masked language modeling (MLM).

Pre-training tasks

Extensions of pre-trained models include Knowledge-Enriched PTMs, Multilingual and Language-Specific PTMs, Multi-Modal PTMs, etc.

Extensions of Pre-trained models

Foundation models are large-scale, pre-trained neural network models developed through self-supervised learning on massive, unlabeled datasets. They are designed to be general-purpose and serve as a base that can be efficiently adapted to a wide variety of specific downstream tasks through methods like fine-tuning or prompting, often eliminating the need to train a new model from scratch.

Foundation Models

While pre-training is a cornerstone of modern NLP, its conceptual origins trace back to the early history of deep learning. Initial strategies involved unsupervised learning for architectures such as RNNs, deep feedforward networks, and autoencoders. The paradigm saw a modern resurgence, driven by the success of large-scale unsupervised learning for word embedding models. Concurrently, a distinct pre-training approach became standard in computer vision, where models were trained on large, labeled datasets like ImageNet before being adapted to other tasks. This diverse history across different domains and techniques set the stage for the large-scale, self-supervised pre-training that now defines the field of NLP.

Historical Context of Pre-training

*Encoder only* -> natural language understanding
- BERT
- RoBERTa (removes NSP task from BERT)

*Decoder only* -> language modeling
- GPT series

*Encoder-Decoder* -> equipped with the ability to perform both natural language understanding and generation
- BART
- T5

Examples of Pre-trained Transformers by Architecture

The adoption of pre-training has caused a significant paradigm shift in the field of Natural Language Processing. This new approach has, in many cases, eliminated the need to conduct large-scale supervised learning for each specific task. Instead, the focus has shifted to adapting general-purpose, pre-trained foundation models to meet the requirements of individual applications.

Paradigm Shift in NLP Driven by Pre-training

Despite the rapid progress driven by large-scale pre-training, the development of truly intelligent systems is still in its early stages. Key open research challenges remain. These include finding methods to learn intelligence more efficiently from smaller, more reasonably sized datasets, and equipping models with the ability to perform complex reasoning and planning. Addressing these issues is crucial for the next wave of advancements in AI.

Future Research Directions in Large-Scale Pre-training

During the pre-training phase, Large Language Models acquire the foundational knowledge necessary for understanding instructions and generating appropriate responses. However, these capabilities exist in a latent state and are not fully functional until they are activated by a subsequent supervisory process, such as fine-tuning.

Role of Pre-training in Developing Latent Abilities

The pre-training of large language models relies on vast and varied text corpora. Key sources for these datasets include webpages, books, conversational text, software code, Wikipedia, and news articles, in addition to other materials like scientific papers and content from question-and-answer (Q&A) platforms.

Common Data Sources for Pre-training LLMs

A training methodology where the parameters of a pre-trained Transformer model are held constant, or 'frozen'. During this process, only a set of newly introduced, learnable parameters are updated. This approach allows for model adaptation while preserving the original, powerful representations of the base Transformer.

Training Auxiliary Parameters with a Fixed Transformer Model

The combination of advanced neural sequence architectures, particularly the Transformer, with large-scale self-supervised learning techniques has been a pivotal development in AI. This synergy is what unlocked the potential for creating universal models capable of both language understanding and generation.

Synergy of Transformers and Self-Supervised Learning

Pre-training in Natural Language Processing primarily addresses two categories of problems: sequence modeling, also referred to as sequence encoding, and sequence generation. Despite their distinct forms, these two problem types can be conceptually unified and described using a single, general model formulation for the sake of simplicity.

Core Problem Types in NLP Pre-training

Introductory discussions on pre-training are intentionally limited in scope and do not cover all aspects of the topic. To maintain focus, such treatments often omit detailed explorations of advanced subjects. For instance, the full range of fine-tuning methods for adapting models to diverse situations, as well as the topic of large language models—despite their significance in AI—are typically deferred to more advanced discussions.

Scope of Introductory Discussions on Pre-training

Self-supervised pre-training is a versatile approach that can be applied to various neural network architectures. This includes encoder-only models, decoder-only models, and full encoder-decoder structures, allowing for the development of foundation models tailored to different types of NLP tasks.

Application of Self-Supervised Pre-training Across Model Architectures

The initial examination of the pre-training paradigm focuses on the fundamental ideas for addressing its two main challenges: optimizing a model for broad generalizability and adapting it effectively to specific downstream applications.

Scope of Foundational Concepts in Pre-training and Adaptation

In Natural Language Processing, tokens are the fundamental units of text created through a process called tokenization. While the terms 'token' and 'word' are often used interchangeably for simplicity, they have distinct meanings, as tokens are the basic building blocks that models process and may not always correspond directly to words.

Tokens vs. Words in NLP

Self-supervised pre-training is a paradigm where a model learns from vast quantities of unlabeled data by creating its own supervisory signals directly from the input. For example, it might learn by predicting masked or corrupted parts of the data. Following this pre-training phase, the model is adapted for specific downstream applications, which can be accomplished through methods like supervised fine-tuning on labeled datasets or via prompting to enable zero-shot or few-shot learning.

Self-supervised Pre-training

A fundamental distinction between pre-training and fine-tuning lies in the scale of data required. While more fine-tuning data is generally beneficial, the amount needed is orders of magnitude smaller than what is required for pre-training. For instance, fine-tuning can be effectively performed with tens or hundreds of thousands of samples, or even fewer if the data is of high quality. In contrast, pre-training models demands billions or even trillions of tokens, which consequently results in significantly larger computational requirements and longer training times.

Data Scale Disparity: Pre-training vs. Fine-tuning

A small biotech company wants to build an AI model to classify protein sequences for a very specific function. They have a high-quality, but small, labeled dataset of 10,000 sequences. They have limited computational resources and a tight deadline. Which of the following strategies represents the most effective and efficient approach for them to develop a high-performing model?

Based on the standard two-stage paradigm for building modern language models, analyze the fundamental flaw in the startup's approach described in the case study. Explain how they should have used both datasets to build a more effective chatbot.

Diagnosing a Flawed Model Development Strategy

The development of large-scale AI models typically involves two distinct stages. Match each characteristic below to the stage it describes.

An introductory overview of pre-training in Natural Language Processing (NLP) typically focuses on the foundational concepts, such as self-supervised pre-training and its application across different model architectures. Because of the vast scope of the field, more advanced topics—including various methods for fine-tuning pre-trained models and detailed examinations of large language models (LLMs)—are generally deferred to subsequent, specialized discussions.

Learn Before

Related

Learn After