1. Addition-based methods: Introduce additional trainable neural modules or parameters that do not exist in the original model. Common addition-based methods include Prefix Tuning, Prompt Tuning.

2. Specification-based methods: Specifies that certain parameters in the original model become trainable, while others are frozen.

3. Reparameterization-based methods: Reparameterize an existing optimization procedure into a parameter-efficient form by transforming it.

University of Illinois at Urbana-Champaign

Google

A more efficient method for fine-tuning parameters of very large-scale models. 
\By fine-tuning only a fraction of the parameters of a very large model, the cost of computation and storage is greatly reduced.

Delta Tuning

Ding, Ning et al. “Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.” ArXiv abs/2203.06904 (2022): n. pag. 
From  https://arxiv.org/pdf/2203.06904.pdf

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

1. Delta Tuning enables faster training of large models, and the scale of its backpropagation is significantly reduced, reducing the computational cost.

2. Delta Tuning can improve the multi-task learning ability of the model and keep the additional storage relatively low.

3. Delta Tuning can alleviate the catastrophic forgetting problem in the model. 

4. Delta Tuning enables language modeling as a service and enables in-batch parallel computing.

Learn Before

Related