Similarly to tuning-free prompting, it can retain knowledge in LMs and is suitable in few-shot scenarios. Often superior accuracy to tuning-free prompting.

University of Michigan - Ann Arbor

1. Promptless Fine-tuning
2. Tuning-free Prompting
3. Fixed-LM Prompt Tuning
4. Fixed-Prompt LM Tuning
5. Prompt+LM Fine-tuning

Major Tuning Strategy Types

https://arxiv.org/pdf/2107.13586.pdf
It's one amazing survey paper proposing existing prompt-tuning methods.

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing


Simplicity, no need for prompt design. Tuning all the LM parameters allows the model to fit to larger training datasets.

Advantages of Promptless Fine-tuning

 LMs may overfit or not learn stably on smaller datasets.

Disadvantages of Promptless Fine-tuning

Efficiency, there is no parameter update process. No catastrophic forgetting, as LM parameters remain fixed. Applicable in zero-shot settings.

Advantages of Tuning-free Prompting

Because prompts are the only method that provides the task specification, heavy engineering is necessary to achieve high accuracy. In particular, in the in-context learning setting, providing many answered prompts can be slow at test time, and thus cannot easily use large training datasets.

Disadvantages of Tuning-free Prompting

Advantages of Fixed-LM Prompt Tuning

Not applicable in zero-shot scenarios. While effective in few-shot scenarios, representation power is limited in large-data settings. Prompt engineering through the choice of hyperparameters or seed prompts is necessary. Prompts are usually not human-interpretable or manipulable.

Disadvantages of Fixed-LM Prompt Tuning

Prompt or answer engineering more completely specifies the task, allowing for more efficient learning, particularly in few-shot scenarios.

Advantages of Fixed-prompt LM Tuning

Prompt or answer engineering are still required, although perhaps not as much as without prompting. LMs fine-tuned on one downstream task may not be effective on another one.

Disadvantages of Fixed-prompt LM Tuning

This is the most expressive method, likely suitable for high-data settings.

Advantages of Prompt+LM Tuning

Requires training and storing all parameters of the models. May overfit to small datasets.

Learn Before

Related