Google

A fundamental issue following the pre-training phase is applying the generalized pre-trained model, denoted as $$g_{\hat{\theta}}(\cdot)$$, to specific downstream tasks. To successfully adapt the model to these downstream tasks, it is necessary to slightly adjust its parameters, denoted as $$\hat{\theta}$$, using labeled data, or alternatively, to prompt the model with task descriptions.

Applying and Adapting Pre-trained Models to Downstream Tasks

- Choose appropriate pre-training task, model architecture and corpus. Currently, the language model is the most popular pretraining task and can more efficiently solve a wide range of NLP problems. However, different pre-training tasks have their own bias and give different effects for different tasks. Besides, the data distribution of the downstream task should be approximate to PTMs.

- Choose appropriate layers. Given a pre-trained deep model, different layers should capture different kinds of information, such as POS tagging, parsing, long-term dependencies, semantic roles, coreference. 

- There are two common ways of model transfer: feature extraction (where the pre-trained parameters are frozen), and fine-tuning (where the pre-trained parameters are unfrozen and fine-tuned). In feature extraction way, the pre-trained models are regarded as off-the-shelf feature extractors. Moreover, it is important to expose the internal layers as they typically encode the most transferable representations.

Transfer knowledge of a PTM to the downstream NLP tasks

Besides standard fine-tuning, there are also some useful fine-tuning strategies.

- Two-stage fine-tuning: It introduces an intermediate stage between pretraining and fine-tuning. In the first stage, the PTM is transferred into a model fine-tuned by an intermediate task or corpus. In the second stage, the transferred model is fine-tuned to the target task.

- Multi-task fine-tuning: Fine-tuning the model across multiple tasks allows sharing information between the different tasks and positive transfer to other related tasks.

- Fine-tuning with extra adaptation modules: The main drawback of fine-tuning is its parameter inefficiency: every downstream task has its own fine-tuned parameters. Therefore, a better solution is to inject some fine-tunable adaptation modules into PTMs while the original parameters are fixed.

Fine-Tuning Strategies

Pre-trained models (PTMs) like BERT are generally well-suited for a variety of downstream NLP tasks. After a model has been fine-tuned for a specific purpose, it can be applied to that task. Some classic examples of such applications include:

- General Evaluation Benchmark

- Question Answering

- Sentiment Analysis

- Named Entity Recognition

- Machine Translation

Applications of PTMs

Fine-tuning is a prevalent technique for adapting a pre-trained sequence encoding model for a specific application. The process begins with an encoder, such as a standard Transformer encoder, which is denoted as $Encode_{\theta}(\cdot)$ with parameters $\theta$. After this model has been pre-trained to find its optimal parameters, denoted as $\hat{\theta}$, it can be used to process any input sequence and generate its corresponding numerical representation.

Fine-tuning for Sequence Encoding Models

Adapting a pre-trained model for a new task involves combining it with a new prediction network. The subsequent fine-tuning is a standard optimization process that starts by initializing the model with its pre-trained parameters, denoted as $$\hat{\theta}$$. The entire model, including the new network's parameters $$\omega$$, is then trained on a task-specific labeled dataset. This supervised process minimizes a loss function to produce optimized parameters, $$\tilde{\theta}$$ and $$\tilde{\omega}$$, thereby specializing the model for the new task.

Fine-Tuning Pre-trained Models for Downstream Tasks

As an alternative to full fine-tuning, a classifier can be efficiently adapted to work in tandem with a pre-trained encoder by freezing the encoder's parameters, $$ \hat{\theta} $$. This maintains their pre-trained state, allowing the optimization process to focus solely on updating the classifier's parameters, $$ \omega $$, for the specific task.

Freezing Encoder Parameters During Fine-Tuning

After a model's parameters have been optimized during pre-training, the subsequent step in adapting the model is to remove the pre-training-specific output layer, denoted by the parameters $$\widehat{\mathbf{W}}$$. This layer is discarded because it is exclusively tailored to the pre-training objective and is not applicable to downstream tasks. Dropping it leaves the core pre-trained encoder, parameterized by $$\hat{\theta}$$, ready to be either further fine-tuned or directly applied as a fixed feature extractor for new applications.

Discarding the Pre-training Head for Downstream Adaptation

To adapt a pre-trained model for a specific downstream task, instructions can be provided in textual form as part of the input sequence. These instructions can vary in format, ranging from a concise task name used as a prefix to a more comprehensive description of the task's requirements.

Textual Instructions for Task Adaptation

The specific requirements of a downstream task dictate key aspects of the model's design. This includes defining the expected input and output formats and determining the appropriate architecture for the prediction network that is added to the pre-trained model.

Influence of Downstream Task on Model Architecture

Fine-tuning is a crucial and widely utilized technique in the development of Large Language Models, with applications that extend far beyond specific use cases like instruction following. It serves as a fundamental method for adapting LLMs for a diverse range of specialized tasks.

Broad Applications of Fine-Tuning in LLM Development

Initial explorations of fine-tuning typically provide a foundational overview of its core processes, while in-depth discussions on the extensive research and advanced methods are often reserved for more specialized, subsequent sections.

Scope of Introductory Fine-Tuning Discussion

LLM alignment refers to the process of guiding a Large Language Model to behave in a manner that is consistent with human intentions. This ensures the model's outputs and actions are desirable and appropriate.

LLM Alignment

The two-stage paradigm for Transformer encoder models consists of a pre-training phase and an application phase. During pre-training, the encoder is paired with a Softmax layer and trained using self-supervision to develop general language representations. During the subsequent application phase, the initial Softmax layer is discarded, and the pre-trained encoder is combined with a task-specific prediction network. To ensure optimal performance on specialized downstream tasks, this combined system undergoes a fine-tuning process using labeled data.

Pre-train and Fine-tune Paradigm for Encoder Models

Adapting a pre-trained model for a specific downstream application generally requires a fine-tuning process. While pre-training equips a model with a broad understanding of language, this general knowledge is often insufficient for specialized tasks, necessitating further training on task-specific data to achieve desired performance.

Necessity of Fine-Tuning for Downstream Task Adaptation

Fine-tuning has become a de facto standard research method for adapting Large Language Models. This approach leverages the token prediction capabilities developed during pre-training, generalizing them to enable the model to perform new, specific tasks.

Fine-Tuning as a Standard Adaptation Method for LLMs

Prompting is a technique for guiding a pre-trained language model to perform a specific task by structuring its input as a textual instruction or query. This method leverages the model's existing knowledge to generate desired outputs without needing to update its parameters through retraining. It is a foundational mechanism that enables advanced application strategies such as zero-shot and few-shot learning.

Prompting in Language Models

The pre-training and fine-tuning paradigm operates on the principle that LLMs acquire latent abilities for instruction comprehension and response generation during pre-training. However, these learned instruction-response mappings may not have a high probability of being generated during inference. Fine-tuning serves as a mechanism to activate these dormant capabilities by slightly adjusting the model's parameters using a small set of supervised data, which increases the likelihood of generating the desired responses to instructions.

Fine-Tuning as a Mechanism for Activating Pre-Trained Knowledge

A startup wants to adapt a large, pre-trained language model to classify customer sentiment (positive, negative, neutral). They have a very small labeled dataset (fewer than 500 examples) and extremely limited access to high-performance computing, making extensive retraining financially unfeasible. Which adaptation approach is most suitable for their situation?

Adapting Large Language Models through prompting is a highly efficient process because it does not require any additional training or parameter tuning once the model has been developed. This allows for rapid and cost-effective customization of LLMs for various tasks without altering the underlying model.

Efficiency of LLM Adaptation via Prompting

A developer intends to specialize a general-purpose, pre-trained language model for a new text classification task by updating its internal parameters. Arrange the following steps in the correct chronological order to accomplish this adaptation.

A company has a single, powerful, general-purpose language model. They need to adapt this model for two distinct new features. For each feature described below, recommend one of the two primary adaptation approaches (adjusting the model's internal weights using task-specific data, OR guiding the model with carefully designed textual inputs). Justify your choice for each feature based on its specific goals and constraints.

Learn Before

Related