When the model is trained for a series of tasks in a certain order, without regularization, updating the full parameters of the pre-trained model can lead to catastrophic forgetting. Delta Tuning, on the other hand, only requires fine-tuning of a small number of parameters, so it may be a solution to catastrophic forgetting.

University of Illinois at Urbana-Champaign

Google

1. Delta Tuning enables faster training of large models, and the scale of its backpropagation is significantly reduced, reducing the computational cost.

2. Delta Tuning can improve the multi-task learning ability of the model and keep the additional storage relatively low.

3. Delta Tuning can alleviate the catastrophic forgetting problem in the model. 

4. Delta Tuning enables language modeling as a service and enables in-batch parallel computing.

Advantages of Delta Tuning

Ding, Ning et al. “Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Models.” ArXiv abs/2203.06904 (2022): n. pag. 
From  https://arxiv.org/pdf/2203.06904.pdf

Learn Before

Related