In soft parameter sharing, each task has its own model with its own parameters. The distance between the parameters of the model is then regularized in order to encourage the parameters to be similar. The constraints used for soft parameter sharing in deep neural networks have been greatly inspired by regularization techniques for MTL that have been developed for other models.

University of Michigan - Ann Arbor

1) Hard Parameter Sharing

2) Soft Parameter Sharing

MTL Methods for Deep Learning

An Overview of Multi-Task Learning in Deep Neural Networks

Hard parameter sharing is the most commonly used approach to MTL in neural networks. It is generally applied by sharing the hidden layers between all tasks, while keeping several task-specific output layers. Hard parameter sharing greatly reduces the risk of overfitting. In fact, the risk of overfitting the shared parameters is an order N, where N is the number of tasks, smaller than overfitting the task-specific parameters, the output layers. Intuitively, this means the more tasks we are learning simultaneously, the more our model has to find a representation that captures all of the tasks and the less is our chance of overfitting on our original task.

Learn Before

Related