When the model is trained for a series of tasks in a certain order, without regularization, updating the full parameters of the pre-trained model can lead to catastrophic forgetting. Delta Tuning, on the other hand, only requires fine-tuning of a small number of parameters, so it may be a solution to catastrophic forgetting.

Why Delta Tuning can alleviate the catastrophic forgetting problem in models？

1. Delta Tuning enables faster training of large models, and the scale of its backpropagation is significantly reduced, reducing the computational cost.

2. Delta Tuning can improve the multi-task learning ability of the model and keep the additional storage relatively low.

3. Delta Tuning can alleviate the catastrophic forgetting problem in the model. 

4. Delta Tuning enables language modeling as a service and enables in-batch parallel computing.

University of Illinois at Urbana-Champaign

A more efficient method for fine-tuning parameters of very large-scale models. 
\By fine-tuning only a fraction of the parameters of a very large model, the cost of computation and storage is greatly reduced.

Delta Tuning

Ding, Ning et al. “Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.” ArXiv abs/2203.06904 (2022): n. pag. 
From  https://arxiv.org/pdf/2203.06904.pdf

Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models

Advantages of Delta Tuning

1. Addition-based methods: Introduce additional trainable neural modules or parameters that do not exist in the original model. Common addition-based methods include Prefix Tuning, Prompt Tuning.

2. Specification-based methods: Specifies that certain parameters in the original model become trainable, while others are frozen.

3. Reparameterization-based methods: Reparameterize an existing optimization procedure into a parameter-efficient form by transforming it.

Learn Before

Related

Learn After